2024-05-06T03:31:14.355 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-06T03:31:14.355 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi/7690146 --name yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} -- /home/teuthworker/archive/yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi/7690146/orig.config.yaml 2024-05-06T03:31:14.378 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T03:31:14.453 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi/7690146 branch: wip-yuriw-testing-20240503.181540-main description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: yweinste@redhat.com first_in_suite: false job_id: '7690146' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuriw-testing-20240503.181540-main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e08cbbf6fd79a8dd2949f340bce25e28ea3db004 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e08cbbf6fd79a8dd2949f340bce25e28ea3db004 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-yuriw-testing-20240503.181540-main sha1: e08cbbf6fd79a8dd2949f340bce25e28ea3db004 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4116 sha1: e08cbbf6fd79a8dd2949f340bce25e28ea3db004 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuriw-testing-20240503.181540-main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e08cbbf6fd79a8dd2949f340bce25e28ea3db004 targets: smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvpA7ozbIen0YWnvxJtzldi8hpjW4oUapdfSZmK2PG8nBJSlU4oFVyMh9agfnL8WGEG9PnvTY3zz0hj1NmCinc= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_23:44:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-06T03:31:14.453 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa; will attempt to use it 2024-05-06T03:31:14.454 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks 2024-05-06T03:31:14.454 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-06T03:31:14.456 INFO:teuthology.task.internal:Checking packages... 2024-05-06T03:31:14.488 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e08cbbf6fd79a8dd2949f340bce25e28ea3db004' 2024-05-06T03:31:14.489 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-06T03:31:14.489 INFO:teuthology.packaging:ref: None 2024-05-06T03:31:14.489 INFO:teuthology.packaging:tag: None 2024-05-06T03:31:14.489 INFO:teuthology.packaging:branch: wip-yuriw-testing-20240503.181540-main 2024-05-06T03:31:14.489 INFO:teuthology.packaging:sha1: e08cbbf6fd79a8dd2949f340bce25e28ea3db004 2024-05-06T03:31:14.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuriw-testing-20240503.181540-main 2024-05-06T03:31:14.712 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3516.ge08cbbf6 2024-05-06T03:31:14.714 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-06T03:31:14.721 INFO:teuthology.task.internal:no buildpackages task found 2024-05-06T03:31:14.721 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-06T03:31:14.762 INFO:teuthology.task.internal:Saving configuration 2024-05-06T03:31:14.772 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-06T03:31:14.800 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T03:31:14.824 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi/7690146', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 03:25:24.627477', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvpA7ozbIen0YWnvxJtzldi8hpjW4oUapdfSZmK2PG8nBJSlU4oFVyMh9agfnL8WGEG9PnvTY3zz0hj1NmCinc='} 2024-05-06T03:31:14.824 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-06T03:31:14.842 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-05-06T03:31:14.842 INFO:teuthology.run_tasks:Running task console_log... 2024-05-06T03:31:14.946 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2f9706df70>, signals=[15]) 2024-05-06T03:31:14.946 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-06T03:31:14.976 INFO:teuthology.task.internal:Opening connections... 2024-05-06T03:31:14.976 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-05-06T03:31:14.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T03:31:15.056 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-06T03:31:15.084 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-05-06T03:31:15.149 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-05-06T03:31:15.149 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-05-06T03:31:15.206 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-05-06T03:31:15.206 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-05-06T03:31:15.206 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-05-06T03:31:15.206 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-05-06T03:31:15.206 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-05-06T03:31:15.206 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T03:31:15.207 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T03:31:15.208 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-05-06T03:31:15.226 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-06T03:31:15.234 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-06T03:31:15.381 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-06T03:31:15.381 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-05-06T03:31:15.399 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-06T03:31:15.481 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-06T03:31:15.481 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-05-06T03:31:15.585 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-06T03:31:15.621 INFO:teuthology.run_tasks:Running task kernel... 2024-05-06T03:31:15.635 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-06T03:31:15.635 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-06T03:31:15.635 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T03:31:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T03:31:15.652 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-05-06T03:31:15.710 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-437.el9.x86_64 2024-05-06T03:31:15.710 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-437.el9.x86_64 2024-05-06T03:31:15.711 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-05-06T03:31:18.223 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.0 MB 00:00 2024-05-06T03:31:25.548 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-05-06T03:31:30.940 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.2 MB 00:01 2024-05-06T03:31:32.418 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 116 kB/s | 16 kB 00:00 2024-05-06T03:31:33.234 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-05-06T03:31:38.350 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-06T03:31:40.231 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T03:31:40.232 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T03:31:40.316 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:31:40.317 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:31:40.317 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:31:40.448 DEBUG:teuthology.orchestra.run.smithi163:> echo no | sudo yum reinstall kernel || true 2024-05-06T03:31:40.954 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:02 ago on Mon 06 May 2024 03:31:38 AM UTC. 2024-05-06T03:31:41.086 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:31:41.094 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:31:41.094 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-05-06T03:31:41.094 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:31:41.094 INFO:teuthology.orchestra.run.smithi163.stdout:Reinstalling: 2024-05-06T03:31:41.094 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 255 k 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 0 2024-05-06T03:31:41.095 INFO:teuthology.orchestra.run.smithi163.stderr:Operation aborted. 2024-05-06T03:31:41.158 INFO:teuthology.orchestra.run.smithi163.stdout:Is this ok [y/N]: 2024-05-06T03:31:41.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum reinstall -y kernel || true 2024-05-06T03:31:41.655 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 03:31:38 AM UTC. 2024-05-06T03:31:41.774 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:31:41.774 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:Reinstalling: 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 255 k 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 0 2024-05-06T03:31:41.775 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:31:42.486 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-437.el9.x86_64.rpm 573 kB/s | 255 kB 00:00 2024-05-06T03:31:42.486 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:31:42.486 INFO:teuthology.orchestra.run.smithi163.stdout:Total 359 kB/s | 255 kB 00:00 2024-05-06T03:31:42.505 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:31:42.717 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:31:42.718 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:31:42.725 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:31:42.725 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:31:42.864 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:31:42.945 INFO:teuthology.orchestra.run.smithi163.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T03:31:42.945 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T03:31:43.611 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T03:31:43.611 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T03:31:44.330 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T03:31:44.331 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:31:44.331 INFO:teuthology.orchestra.run.smithi163.stdout:Reinstalled: 2024-05-06T03:31:44.331 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T03:31:44.331 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:31:44.331 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:31:44.517 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T03:31:44.604 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T03:31:44.604 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T03:31:44.604 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T03:31:44.604 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-06T03:31:44.604 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T03:31:44.637 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-05-06T03:31:44.659 DEBUG:teuthology.parallel:result is None 2024-05-06T03:31:44.660 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-06T03:31:44.667 INFO:teuthology.task.internal:Creating test directory... 2024-05-06T03:31:44.667 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T03:31:44.714 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-06T03:31:44.724 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-06T03:31:44.827 INFO:teuthology.task.internal:Creating archive directory... 2024-05-06T03:31:44.827 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T03:31:44.851 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-06T03:31:44.876 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-06T03:31:44.877 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T03:31:44.914 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:31:44.923 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:31:44.925 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-06T03:31:44.933 INFO:teuthology.task.internal:Configuring sudo... 2024-05-06T03:31:44.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T03:31:44.989 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-06T03:31:44.998 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-06T03:31:44.999 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T03:31:45.044 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T03:31:45.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T03:31:45.181 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T03:31:45.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T03:31:45.301 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:31:45.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T03:31:45.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-05-06T03:31:45.443 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T03:31:45.834 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-06T03:31:45.841 INFO:teuthology.task.internal:Starting timer... 2024-05-06T03:31:45.842 INFO:teuthology.run_tasks:Running task pcp... 2024-05-06T03:31:45.885 INFO:teuthology.run_tasks:Running task selinux... 2024-05-06T03:31:45.920 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-06T03:31:45.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-05-06T03:31:45.973 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-05-06T03:31:45.975 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-06T03:31:45.975 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-05-06T03:31:46.006 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-05-06T03:31:46.006 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-05-06T03:31:46.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-06T03:31:46.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T03:31:46.070 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-06T03:31:46.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-05-06T03:31:46.135 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-06T03:31:46.144 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-06T03:31:46.161 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-06T03:31:46.163 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-06T03:35:33.110 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-05-06T03:35:33.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-05-06T03:35:33.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T03:35:33.189 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-05-06T03:35:33.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-05-06T03:35:33.274 INFO:teuthology.run_tasks:Running task clock... 2024-05-06T03:35:33.285 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-06T03:35:33.285 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T03:35:33.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T03:35:33.341 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T03:35:33.354 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T03:35:33.377 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-05-06T03:35:33.387 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-05-06T03:35:33.400 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T03:35:33.413 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T03:35:33.467 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-05-06T03:35:33.470 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T03:35:33.470 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-05-06T03:35:33.470 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T03:35:33.471 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T03:35:33.471 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T03:35:33.471 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T03:35:33.471 INFO:teuthology.run_tasks:Running task install... 2024-05-06T03:35:33.480 DEBUG:teuthology.task.install:project ceph 2024-05-06T03:35:33.480 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e08cbbf6fd79a8dd2949f340bce25e28ea3db004'}} 2024-05-06T03:35:33.480 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e08cbbf6fd79a8dd2949f340bce25e28ea3db004'} 2024-05-06T03:35:33.480 INFO:teuthology.task.install:Using flavor: default 2024-05-06T03:35:33.487 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-06T03:35:33.487 INFO:teuthology.task.install:extra packages: [] 2024-05-06T03:35:33.488 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e08cbbf6fd79a8dd2949f340bce25e28ea3db004', 'tag': None, 'wait_for_package': False} 2024-05-06T03:35:33.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 2024-05-06T03:35:33.647 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuriw-testing-20240503.181540-main/e08cbbf6fd79a8dd2949f340bce25e28ea3db004/centos/9/flavors/default/ 2024-05-06T03:35:33.647 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3516.ge08cbbf6 2024-05-06T03:35:33.761 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw-testing-20240503.181540-main/e08cbbf6fd79a8dd2949f340bce25e28ea3db004/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw-testing-20240503.181540-main/e08cbbf6fd79a8dd2949f340bce25e28ea3db004/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw-testing-20240503.181540-main/e08cbbf6fd79a8dd2949f340bce25e28ea3db004/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-06T03:35:33.762 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:35:33.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-06T03:35:33.787 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-05-06T03:35:33.787 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e08cbbf6fd79a8dd2949f340bce25e28ea3db004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-06T03:35:33.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-06T03:35:33.926 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-06T03:35:33.989 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-05-06T03:35:33.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-05-06T03:35:34.217 INFO:teuthology.orchestra.run.smithi163.stdout:50 files removed 2024-05-06T03:35:34.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-05-06T03:35:35.213 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 118 kB/s | 84 kB 00:00 2024-05-06T03:35:35.573 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-05-06T03:35:35.923 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-06T03:35:36.706 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-05-06T03:35:38.483 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-05-06T03:35:42.492 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-06T03:35:43.984 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-05-06T03:35:48.627 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 21 MB 00:04 2024-05-06T03:35:53.819 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-05-06T03:35:56.003 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:35:56.005 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:35:56.005 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:35:56.005 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:35:56.005 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:35:56.005 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 11 M 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 3.4 M 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 3.1 M 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 5.4 M 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 21 M 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 25 k 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 747 k 2024-05-06T03:35:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 500 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 5.4 M 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 45 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 135 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 161 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-06T03:35:56.007 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 320 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 298 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 99 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout:Install 30 Packages 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-05-06T03:35:56.008 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:35:56.009 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 60 M 2024-05-06T03:35:56.009 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:35:56.903 INFO:teuthology.orchestra.run.smithi163.stdout:(1/32): ceph-base-19.0.0-3516.ge08cbbf6.el9.x86 11 MB/s | 5.4 MB 00:00 2024-05-06T03:35:56.953 INFO:teuthology.orchestra.run.smithi163.stdout:(2/32): ceph-selinux-19.0.0-3516.ge08cbbf6.el9. 507 kB/s | 25 kB 00:00 2024-05-06T03:35:57.029 INFO:teuthology.orchestra.run.smithi163.stdout:(3/32): libcephfs2-19.0.0-3516.ge08cbbf6.el9.x8 9.7 MB/s | 747 kB 00:00 2024-05-06T03:35:57.087 INFO:teuthology.orchestra.run.smithi163.stdout:(4/32): libradosstriper1-19.0.0-3516.ge08cbbf6. 8.4 MB/s | 500 kB 00:00 2024-05-06T03:35:57.404 INFO:teuthology.orchestra.run.smithi163.stdout:(5/32): librgw2-19.0.0-3516.ge08cbbf6.el9.x86_6 17 MB/s | 5.4 MB 00:00 2024-05-06T03:35:57.446 INFO:teuthology.orchestra.run.smithi163.stdout:(6/32): python3-ceph-argparse-19.0.0-3516.ge08c 1.1 MB/s | 45 kB 00:00 2024-05-06T03:35:57.488 INFO:teuthology.orchestra.run.smithi163.stdout:(7/32): python3-ceph-common-19.0.0-3516.ge08cbb 3.2 MB/s | 135 kB 00:00 2024-05-06T03:35:57.772 INFO:teuthology.orchestra.run.smithi163.stdout:(8/32): python3-cephfs-19.0.0-3516.ge08cbbf6.el 569 kB/s | 161 kB 00:00 2024-05-06T03:35:57.839 INFO:teuthology.orchestra.run.smithi163.stdout:(9/32): python3-rados-19.0.0-3516.ge08cbbf6.el9 4.7 MB/s | 320 kB 00:00 2024-05-06T03:35:57.897 INFO:teuthology.orchestra.run.smithi163.stdout:(10/32): python3-rbd-19.0.0-3516.ge08cbbf6.el9. 5.0 MB/s | 298 kB 00:00 2024-05-06T03:35:57.964 INFO:teuthology.orchestra.run.smithi163.stdout:(11/32): python3-rgw-19.0.0-3516.ge08cbbf6.el9. 1.4 MB/s | 99 kB 00:00 2024-05-06T03:35:58.131 INFO:teuthology.orchestra.run.smithi163.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 642 kB/s | 107 kB 00:00 2024-05-06T03:35:58.198 INFO:teuthology.orchestra.run.smithi163.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-05-06T03:35:58.240 INFO:teuthology.orchestra.run.smithi163.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-06T03:35:58.307 INFO:teuthology.orchestra.run.smithi163.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-05-06T03:35:58.374 INFO:teuthology.orchestra.run.smithi163.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-05-06T03:35:58.417 INFO:teuthology.orchestra.run.smithi163.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-05-06T03:35:58.466 INFO:teuthology.orchestra.run.smithi163.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-06T03:35:58.508 INFO:teuthology.orchestra.run.smithi163.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 859 kB/s | 36 kB 00:00 2024-05-06T03:35:58.592 INFO:teuthology.orchestra.run.smithi163.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-05-06T03:35:58.792 INFO:teuthology.orchestra.run.smithi163.stdout:(21/32): ceph-common-19.0.0-3516.ge08cbbf6.el9. 8.8 MB/s | 21 MB 00:02 2024-05-06T03:35:58.934 INFO:teuthology.orchestra.run.smithi163.stdout:(22/32): ceph-radosgw-19.0.0-3516.ge08cbbf6.el9 4.2 MB/s | 11 MB 00:02 2024-05-06T03:35:59.218 INFO:teuthology.orchestra.run.smithi163.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-05-06T03:35:59.243 INFO:teuthology.orchestra.run.smithi163.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 475 kB/s | 309 kB 00:00 2024-05-06T03:35:59.327 INFO:teuthology.orchestra.run.smithi163.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-05-06T03:35:59.352 INFO:teuthology.orchestra.run.smithi163.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 363 kB/s | 49 kB 00:00 2024-05-06T03:35:59.502 INFO:teuthology.orchestra.run.smithi163.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.7 MB/s | 839 kB 00:00 2024-05-06T03:35:59.527 INFO:teuthology.orchestra.run.smithi163.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-05-06T03:35:59.611 INFO:teuthology.orchestra.run.smithi163.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-05-06T03:35:59.653 INFO:teuthology.orchestra.run.smithi163.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-05-06T03:35:59.870 INFO:teuthology.orchestra.run.smithi163.stdout:(31/32): librados2-19.0.0-3516.ge08cbbf6.el9.x8 9.9 MB/s | 3.4 MB 00:00 2024-05-06T03:35:59.978 INFO:teuthology.orchestra.run.smithi163.stdout:(32/32): librbd1-19.0.0-3516.ge08cbbf6.el9.x86_ 8.6 MB/s | 3.1 MB 00:00 2024-05-06T03:35:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:35:59.983 INFO:teuthology.orchestra.run.smithi163.stdout:Total 15 MB/s | 60 MB 00:03 2024-05-06T03:36:00.080 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:36:00.116 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:36:00.116 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:36:00.584 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:36:00.585 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:36:02.159 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:36:02.237 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-06T03:36:02.275 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-06T03:36:02.300 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-06T03:36:02.323 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-06T03:36:02.342 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-06T03:36:02.506 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-06T03:36:02.561 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-06T03:36:02.942 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-06T03:36:02.994 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 7/34 2024-05-06T03:36:03.070 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 7/34 2024-05-06T03:36:03.158 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-3516.ge08cbbf6.el9.x86_64 8/34 2024-05-06T03:36:03.187 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 9/34 2024-05-06T03:36:03.244 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 9/34 2024-05-06T03:36:03.283 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-06T03:36:03.332 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-3516.ge08cbbf6.el9. 11/34 2024-05-06T03:36:03.412 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-3516.ge08cbbf6.el9.x86_64 12/34 2024-05-06T03:36:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-3516.ge08cbbf6.el9.x86_6 13/34 2024-05-06T03:36:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3516.ge08cbbf6.el9.x86_6 13/34 2024-05-06T03:36:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-06T03:36:03.632 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-06T03:36:03.664 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-06T03:36:03.704 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-06T03:36:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-06T03:36:03.758 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-06T03:36:03.791 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-06T03:36:04.113 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-06T03:36:04.154 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-3516.ge08cbbf6.el9.x86_64 22/34 2024-05-06T03:36:04.244 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-3516.ge08cbbf6.el9.x86_64 22/34 2024-05-06T03:36:04.275 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 23/34 2024-05-06T03:36:04.311 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-06T03:36:04.791 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-06T03:36:04.815 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 26/34 2024-05-06T03:36:04.886 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 26/34 2024-05-06T03:36:04.956 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 27/34 2024-05-06T03:36:04.992 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x8 28/34 2024-05-06T03:36:07.212 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x86_64 29/34 2024-05-06T03:36:07.247 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x86_64 29/34 2024-05-06T03:36:07.791 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x86_64 29/34 2024-05-06T03:36:07.834 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-3516.ge08cbbf6.el9.x86_64 30/34 2024-05-06T03:36:07.978 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-3516.ge08cbbf6.el9.x86_64 30/34 2024-05-06T03:36:07.979 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-06T03:36:07.979 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-06T03:36:07.979 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:08.011 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-3516.ge08cbbf6.el9.x86_64 31/34 2024-05-06T03:36:17.587 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3516.ge08cbbf6.el9.x86_64 31/34 2024-05-06T03:36:17.587 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-05-06T03:36:17.588 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:18.595 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 32/34 2024-05-06T03:36:18.620 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 32/34 2024-05-06T03:36:18.621 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:36:18.621 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-06T03:36:18.621 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-06T03:36:18.621 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-06T03:36:18.621 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:18.624 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-06T03:36:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-06T03:36:18.732 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-06T03:36:19.902 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 3/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-3516.ge08cbbf6.el9.x86_64 4/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 5/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-3516.ge08cbbf6.el9.x86_6 6/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 7/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3516.ge08cbbf6.el9. 8/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x8 9/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-3516.ge08cbbf6.el9.x86_64 10/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-3516.ge08cbbf6.el9.x86_64 11/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 12/34 2024-05-06T03:36:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 13/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-06T03:36:19.904 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 31/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-06T03:36:19.905 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-3516.ge08cbbf6.el9.x86_64 33/34 2024-05-06T03:36:20.668 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-06T03:36:20.669 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:20.671 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:20.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-05-06T03:36:21.542 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:28 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:21.664 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 49 M 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 163 k 2024-05-06T03:36:21.665 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:21.666 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 50 M 2024-05-06T03:36:21.667 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 211 M 2024-05-06T03:36:21.667 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:36:22.014 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): libcephsqlite-19.0.0-3516.ge08cbbf6.el9. 673 kB/s | 163 kB 00:00 2024-05-06T03:36:22.182 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 742 kB/s | 305 kB 00:00 2024-05-06T03:36:22.217 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 314 kB/s | 64 kB 00:00 2024-05-06T03:36:24.493 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): ceph-test-19.0.0-3516.ge08cbbf6.el9.x86_ 18 MB/s | 49 MB 00:02 2024-05-06T03:36:24.494 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:36:24.495 INFO:teuthology.orchestra.run.smithi163.stdout:Total 18 MB/s | 50 MB 00:02 2024-05-06T03:36:24.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:36:24.543 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:36:24.543 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:36:24.912 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:36:24.913 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:36:25.440 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:36:25.510 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-06T03:36:25.558 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-06T03:36:25.586 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-3516.ge08cbbf6.el9.x86_64 3/4 2024-05-06T03:36:30.241 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3516.ge08cbbf6.el9.x86_64 3/4 2024-05-06T03:36:30.325 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-3516.ge08cbbf6.el9.x86_64 4/4 2024-05-06T03:36:32.691 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:19.0.0-3516.ge08cbbf6.el9.x86_64 4/4 2024-05-06T03:36:32.691 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/4 2024-05-06T03:36:32.691 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2/4 2024-05-06T03:36:32.691 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-06T03:36:32.972 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:32.973 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:33.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-05-06T03:36:33.776 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:40 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:33.903 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 6.6 k 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 2.4 M 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 1.5 M 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 274 k 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 4.7 M 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 17 M 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-06T03:36:33.906 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-06T03:36:33.907 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-06T03:36:33.908 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout:Install 36 Packages 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:33.909 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 30 M 2024-05-06T03:36:33.910 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 108 M 2024-05-06T03:36:33.910 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:36:34.960 INFO:teuthology.orchestra.run.smithi163.stdout:(1/36): ceph-19.0.0-3516.ge08cbbf6.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-05-06T03:36:35.110 INFO:teuthology.orchestra.run.smithi163.stdout:(2/36): ceph-mds-19.0.0-3516.ge08cbbf6.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-05-06T03:36:35.335 INFO:teuthology.orchestra.run.smithi163.stdout:(3/36): ceph-mgr-19.0.0-3516.ge08cbbf6.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2024-05-06T03:36:35.444 INFO:teuthology.orchestra.run.smithi163.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3516.ge08c 2.5 MB/s | 274 kB 00:00 2024-05-06T03:36:35.577 INFO:teuthology.orchestra.run.smithi163.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-05-06T03:36:35.670 INFO:teuthology.orchestra.run.smithi163.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 13 MB/s | 1.2 MB 00:00 2024-05-06T03:36:35.711 INFO:teuthology.orchestra.run.smithi163.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-06T03:36:35.753 INFO:teuthology.orchestra.run.smithi163.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-05-06T03:36:35.795 INFO:teuthology.orchestra.run.smithi163.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-05-06T03:36:35.971 INFO:teuthology.orchestra.run.smithi163.stdout:(10/36): ceph-osd-19.0.0-3516.ge08cbbf6.el9.x86 20 MB/s | 17 MB 00:00 2024-05-06T03:36:35.996 INFO:teuthology.orchestra.run.smithi163.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-05-06T03:36:36.146 INFO:teuthology.orchestra.run.smithi163.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-05-06T03:36:36.171 INFO:teuthology.orchestra.run.smithi163.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 983 kB/s | 172 kB 00:00 2024-05-06T03:36:36.197 INFO:teuthology.orchestra.run.smithi163.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-05-06T03:36:36.222 INFO:teuthology.orchestra.run.smithi163.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-06T03:36:36.247 INFO:teuthology.orchestra.run.smithi163.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-05-06T03:36:36.606 INFO:teuthology.orchestra.run.smithi163.stdout:(17/36): ceph-mon-19.0.0-3516.ge08cbbf6.el9.x86 2.9 MB/s | 4.7 MB 00:01 2024-05-06T03:36:36.856 INFO:teuthology.orchestra.run.smithi163.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-05-06T03:36:36.881 INFO:teuthology.orchestra.run.smithi163.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 238 kB/s | 151 kB 00:00 2024-05-06T03:36:37.040 INFO:teuthology.orchestra.run.smithi163.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 100 kB/s | 43 kB 00:00 2024-05-06T03:36:37.157 INFO:teuthology.orchestra.run.smithi163.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-05-06T03:36:37.174 INFO:teuthology.orchestra.run.smithi163.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-05-06T03:36:37.257 INFO:teuthology.orchestra.run.smithi163.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-05-06T03:36:37.299 INFO:teuthology.orchestra.run.smithi163.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-05-06T03:36:37.358 INFO:teuthology.orchestra.run.smithi163.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-05-06T03:36:37.417 INFO:teuthology.orchestra.run.smithi163.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-05-06T03:36:37.467 INFO:teuthology.orchestra.run.smithi163.stdout:(27/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 282 kB/s | 172 kB 00:00 2024-05-06T03:36:37.484 INFO:teuthology.orchestra.run.smithi163.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2024-05-06T03:36:37.592 INFO:teuthology.orchestra.run.smithi163.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-05-06T03:36:37.634 INFO:teuthology.orchestra.run.smithi163.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 363 kB/s | 79 kB 00:00 2024-05-06T03:36:37.684 INFO:teuthology.orchestra.run.smithi163.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-05-06T03:36:37.701 INFO:teuthology.orchestra.run.smithi163.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 827 kB/s | 90 kB 00:00 2024-05-06T03:36:37.760 INFO:teuthology.orchestra.run.smithi163.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-05-06T03:36:37.835 INFO:teuthology.orchestra.run.smithi163.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-05-06T03:36:37.877 INFO:teuthology.orchestra.run.smithi163.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-05-06T03:36:37.902 INFO:teuthology.orchestra.run.smithi163.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 141 kB/s | 20 kB 00:00 2024-05-06T03:36:37.907 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:36:37.907 INFO:teuthology.orchestra.run.smithi163.stdout:Total 7.6 MB/s | 30 MB 00:03 2024-05-06T03:36:38.043 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:36:38.083 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:36:38.083 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:36:38.379 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:36:38.380 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:36:39.633 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:36:39.705 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-06T03:36:39.738 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-06T03:36:39.766 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-06T03:36:39.836 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-06T03:36:39.890 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-06T03:36:39.922 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-06T03:36:40.014 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-06T03:36:40.095 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-06T03:36:40.142 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-06T03:36:40.177 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-06T03:36:40.207 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-06T03:36:40.241 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-06T03:36:40.274 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-06T03:36:40.315 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-06T03:36:40.358 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-06T03:36:40.393 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-06T03:36:40.431 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-06T03:36:40.505 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-06T03:36:40.643 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-06T03:36:40.723 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-06T03:36:40.783 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-06T03:36:40.823 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-06T03:36:40.871 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-06T03:36:40.926 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-06T03:36:41.121 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-06T03:36:41.169 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-06T03:36:41.246 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-06T03:36:41.335 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-06T03:36:41.376 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-06T03:36:41.502 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-06T03:36:41.663 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3516.ge08cbbf6.el9. 31/36 2024-05-06T03:36:41.686 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-3516.ge08cbbf6.el9.x86_64 32/36 2024-05-06T03:36:41.712 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3516.ge08cbbf6.el9.x86_64 32/36 2024-05-06T03:36:41.712 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:36:41.712 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-06T03:36:41.712 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-06T03:36:41.712 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-06T03:36:41.712 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:43.145 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 33/36 2024-05-06T03:36:43.172 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 33/36 2024-05-06T03:36:43.172 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:36:43.172 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-06T03:36:43.172 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-06T03:36:43.172 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-06T03:36:43.172 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:43.629 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-3516.ge08cbbf6.el9.x86_64 34/36 2024-05-06T03:36:43.652 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-3516.ge08cbbf6.el9.x86_64 34/36 2024-05-06T03:36:43.652 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:36:43.652 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-06T03:36:43.652 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-06T03:36:43.652 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-06T03:36:43.652 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-3516.ge08cbbf6.el9.x86_64 35/36 2024-05-06T03:36:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-3516.ge08cbbf6.el9.x86_64 35/36 2024-05-06T03:36:43.911 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:36:43.911 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-06T03:36:43.911 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-06T03:36:43.911 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-06T03:36:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:43.975 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-3516.ge08cbbf6.el9.x86_64 36/36 2024-05-06T03:36:45.751 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:19.0.0-3516.ge08cbbf6.el9.x86_64 36/36 2024-05-06T03:36:45.751 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/36 2024-05-06T03:36:45.751 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2/36 2024-05-06T03:36:45.751 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-3516.ge08cbbf6.el9.x86_64 3/36 2024-05-06T03:36:45.751 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-3516.ge08cbbf6.el9.x86_64 4/36 2024-05-06T03:36:45.751 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 5/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3516.ge08cbbf6.el9. 6/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-06T03:36:45.752 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-06T03:36:45.753 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-06T03:36:45.753 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-06T03:36:45.753 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-06T03:36:45.753 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-06T03:36:45.753 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-06T03:36:45.753 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-06T03:36:45.754 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-06T03:36:45.755 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:46.750 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-06T03:36:46.751 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-06T03:36:46.752 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:46.753 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:47.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-05-06T03:36:47.626 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:54 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:47.706 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:36:47.747 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:47.748 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:36:47.748 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:47.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-05-06T03:36:48.320 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:55 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:48.441 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:48.442 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:48.442 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:36:48.442 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:48.442 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 763 k 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 763 k 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 769 k 2024-05-06T03:36:48.443 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:36:49.023 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-19.0.0-3516.ge08cbbf6.el9.noarch.rpm 1.3 MB/s | 763 kB 00:00 2024-05-06T03:36:49.023 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:36:49.024 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.3 MB/s | 763 kB 00:00 2024-05-06T03:36:49.024 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:36:49.029 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:36:49.029 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:36:49.035 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:36:49.036 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:36:49.106 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:36:49.565 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/1 2024-05-06T03:36:49.649 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/1 2024-05-06T03:36:50.002 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/1 2024-05-06T03:36:50.230 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/1 2024-05-06T03:36:50.230 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:50.230 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:36:50.230 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:36:50.231 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:50.231 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:50.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-05-06T03:36:50.947 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:57 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:51.070 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:51.071 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:51.071 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-05-06T03:36:51.071 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:51.071 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:36:51.071 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 145 k 2024-05-06T03:36:51.071 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 145 k 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 435 k 2024-05-06T03:36:51.072 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:36:51.295 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-19.0.0-3516.ge08cbb 651 kB/s | 145 kB 00:00 2024-05-06T03:36:51.296 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:36:51.296 INFO:teuthology.orchestra.run.smithi163.stdout:Total 645 kB/s | 145 kB 00:00 2024-05-06T03:36:51.296 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:36:51.304 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:36:51.304 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:36:51.361 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:36:51.361 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:36:51.516 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:36:51.588 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3516.ge08cbbf6. 1/1 2024-05-06T03:36:51.612 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3516.ge08cbbf6. 1/1 2024-05-06T03:36:51.612 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:36:51.612 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-06T03:36:51.612 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:52.487 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3516.ge08cbbf6. 1/1 2024-05-06T03:36:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:52.488 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:36:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:36:52.488 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:52.488 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:52.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-05-06T03:36:53.258 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:00 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:53.341 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:36:53.386 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:53.387 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:36:53.387 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:53.456 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-05-06T03:36:53.969 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:00 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:36:54.090 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:36:54.091 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:54.091 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:36:54.091 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:54.091 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 3.5 M 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 34 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 16 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:54.092 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:36:54.093 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:36:54.093 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-05-06T03:36:54.093 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:54.093 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 8.7 M 2024-05-06T03:36:54.093 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 104 M 2024-05-06T03:36:54.093 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:36:54.598 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3516.ge08 87 kB/s | 16 kB 00:00 2024-05-06T03:36:54.782 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-05-06T03:36:54.824 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 6.3 MB/s | 269 kB 00:00 2024-05-06T03:36:54.891 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): ceph-grafana-dashboards-19.0.0-3516.ge0 71 kB/s | 34 kB 00:00 2024-05-06T03:36:55.041 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-05-06T03:36:55.125 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-05-06T03:36:55.142 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-05-06T03:36:55.242 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-05-06T03:36:55.259 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-05-06T03:36:55.293 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 612 kB/s | 31 kB 00:00 2024-05-06T03:36:55.326 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-05-06T03:36:55.460 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): ceph-mgr-dashboard-19.0.0-3516.ge08cbb 3.4 MB/s | 3.5 MB 00:01 2024-05-06T03:36:55.462 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:36:55.462 INFO:teuthology.orchestra.run.smithi163.stdout:Total 6.3 MB/s | 8.7 MB 00:01 2024-05-06T03:36:55.510 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:36:55.526 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:36:55.526 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:36:55.679 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:36:55.680 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:36:56.218 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:36:56.293 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-06T03:36:56.329 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-06T03:36:56.375 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-06T03:36:56.405 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-06T03:36:56.481 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-06T03:36:56.606 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-06T03:36:56.699 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-06T03:36:56.743 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-06T03:36:56.772 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-06T03:36:56.804 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3516.ge08cbbf6.el9 10/12 2024-05-06T03:36:57.847 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3516.ge08cbbf6.el 11/12 2024-05-06T03:36:57.888 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3516.ge08cbbf6.el9.noa 12/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3516.ge08cbbf6.el9.noa 12/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3516.ge08cbbf6.el 1/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3516.ge08cbbf6.el9.noa 2/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3516.ge08cbbf6.el9 3/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-06T03:36:59.235 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-06T03:36:59.236 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-06T03:36:59.236 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-06T03:36:59.236 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-06T03:36:59.236 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-06T03:36:59.537 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:36:59.538 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:36:59.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-06T03:37:00.302 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:07 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:00.424 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:00.425 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-06T03:37:00.425 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 7.4 M 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-06T03:37:00.426 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-05-06T03:37:00.427 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 38 M 2024-05-06T03:37:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 209 M 2024-05-06T03:37:00.428 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:00.742 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-05-06T03:37:00.792 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.2 MB/s | 803 kB 00:00 2024-05-06T03:37:00.884 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-05-06T03:37:00.943 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 311 kB/s | 18 kB 00:00 2024-05-06T03:37:01.001 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 606 kB/s | 35 kB 00:00 2024-05-06T03:37:01.185 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.6 MB/s | 3.0 MB 00:00 2024-05-06T03:37:01.252 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-05-06T03:37:01.410 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-05-06T03:37:01.469 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-05-06T03:37:01.602 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-05-06T03:37:02.028 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 5.2 MB/s | 7.4 MB 00:01 2024-05-06T03:37:02.186 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-05-06T03:37:02.187 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:02.187 INFO:teuthology.orchestra.run.smithi163.stdout:Total 22 MB/s | 38 MB 00:01 2024-05-06T03:37:02.536 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:02.563 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:02.563 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:02.974 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:02.975 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:03.847 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:03.916 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-06T03:37:03.959 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-06T03:37:04.059 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-06T03:37:04.093 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-06T03:37:04.271 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-06T03:37:04.309 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-06T03:37:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-06T03:37:05.039 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-06T03:37:05.155 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-06T03:37:06.386 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-06T03:37:07.419 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-06T03:37:07.450 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3516.ge08cb 12/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3516.ge08cb 12/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3516.ge08cb 1/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-06T03:37:10.180 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-06T03:37:10.181 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-06T03:37:10.181 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-06T03:37:10.181 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-06T03:37:10.181 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-06T03:37:10.562 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-06T03:37:10.562 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-06T03:37:10.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-06T03:37:10.564 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:10.564 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:10.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-05-06T03:37:11.439 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:18 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:11.561 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:11.562 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:11.562 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 49 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-06T03:37:11.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:11.564 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.2 M 2024-05-06T03:37:11.565 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-05-06T03:37:11.565 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:12.008 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 306 kB/s | 26 kB 00:00 2024-05-06T03:37:12.033 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 176 kB/s | 19 kB 00:00 2024-05-06T03:37:12.150 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-05-06T03:37:12.175 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-05-06T03:37:12.209 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.7 MB/s | 279 kB 00:00 2024-05-06T03:37:12.234 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): ceph-mgr-rook-19.0.0-3516.ge08cbbf6.el9 159 kB/s | 49 kB 00:00 2024-05-06T03:37:12.259 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 54 kB 00:00 2024-05-06T03:37:12.310 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-05-06T03:37:12.335 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-05-06T03:37:12.385 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-06T03:37:12.410 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 1.3 MB/s | 206 kB 00:00 2024-05-06T03:37:12.444 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-05-06T03:37:12.494 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-05-06T03:37:12.495 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:12.495 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-05-06T03:37:12.558 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:12.572 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:12.572 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:12.657 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:12.657 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:13.166 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:13.335 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-06T03:37:13.396 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-06T03:37:13.451 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-06T03:37:13.488 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-06T03:37:13.529 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-06T03:37:13.624 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-06T03:37:13.726 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-06T03:37:13.772 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-06T03:37:14.497 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-06T03:37:14.544 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-06T03:37:14.578 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-06T03:37:14.635 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-06T03:37:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-3516.ge08cbbf6.el9.noarch 13/13 2024-05-06T03:37:15.429 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3516.ge08cbbf6.el9.noarch 13/13 2024-05-06T03:37:15.429 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-06T03:37:15.430 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-06T03:37:15.767 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-06T03:37:15.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-06T03:37:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-06T03:37:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-06T03:37:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-06T03:37:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-06T03:37:15.769 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:15.769 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:16.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-05-06T03:37:16.521 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:23 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:16.641 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:16.642 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:16.642 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:37:16.642 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:16.642 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:16.642 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 142 k 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-05-06T03:37:16.643 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:16.644 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 7.0 M 2024-05-06T03:37:16.644 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 33 M 2024-05-06T03:37:16.644 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:17.270 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-05-06T03:37:17.337 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3516.ge08cbbf6.e 639 kB/s | 142 kB 00:00 2024-05-06T03:37:17.437 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-05-06T03:37:17.462 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-05-06T03:37:17.487 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.5 MB/s | 548 kB 00:00 2024-05-06T03:37:17.513 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 679 kB/s | 51 kB 00:00 2024-05-06T03:37:17.513 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:17.514 INFO:teuthology.orchestra.run.smithi163.stdout:Total 8.0 MB/s | 7.0 MB 00:00 2024-05-06T03:37:17.585 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:17.596 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:17.596 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:17.694 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:17.696 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:17.997 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:18.172 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-06T03:37:18.231 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-06T03:37:18.757 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-06T03:37:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-06T03:37:18.905 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-06T03:37:18.952 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 6/6 2024-05-06T03:37:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 6/6 2024-05-06T03:37:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/6 2024-05-06T03:37:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-06T03:37:19.679 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-06T03:37:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-06T03:37:19.680 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-06T03:37:19.934 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:19.935 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:20.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-05-06T03:37:20.724 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:27 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 890 k 2024-05-06T03:37:20.847 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 970 k 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.8 M 2024-05-06T03:37:20.848 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:21.256 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): ceph-fuse-19.0.0-3516.ge08cbbf6.el9.x86_ 3.0 MB/s | 890 kB 00:00 2024-05-06T03:37:21.648 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 118 kB/s | 80 kB 00:00 2024-05-06T03:37:21.648 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:21.649 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.2 MB/s | 970 kB 00:00 2024-05-06T03:37:21.661 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:21.684 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:21.684 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:21.763 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:21.763 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:21.947 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:22.082 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-06T03:37:22.145 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2/2 2024-05-06T03:37:22.823 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2/2 2024-05-06T03:37:22.824 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/2 2024-05-06T03:37:23.084 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-06T03:37:23.084 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:23.084 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:23.084 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-06T03:37:23.084 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:23.084 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:23.350 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-volume 2024-05-06T03:37:23.871 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:30 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:24.001 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:24.001 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:24.001 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-06T03:37:24.001 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:24.001 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-3516.ge08cbbf6.el9 ceph-noarch 273 k 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-05-06T03:37:24.002 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:24.003 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 350 k 2024-05-06T03:37:24.003 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.5 M 2024-05-06T03:37:24.003 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:24.427 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): ceph-volume-19.0.0-3516.ge08cbbf6.el9.no 715 kB/s | 273 kB 00:00 2024-05-06T03:37:24.735 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 112 kB/s | 77 kB 00:00 2024-05-06T03:37:24.736 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:24.736 INFO:teuthology.orchestra.run.smithi163.stdout:Total 477 kB/s | 350 kB 00:00 2024-05-06T03:37:24.748 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:24.758 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:24.758 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:24.785 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:24.785 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:24.933 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:25.116 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-06T03:37:25.158 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-3516.ge08cbbf6.el9.noarch 2/2 2024-05-06T03:37:25.174 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-3516.ge08cbbf6.el9.noarch 2/2 2024-05-06T03:37:25.174 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:37:25.174 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-06T03:37:25.174 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:25.886 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-3516.ge08cbbf6.el9.noarch 1/2 2024-05-06T03:37:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-06T03:37:26.204 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:26.205 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-3516.ge08cbbf6.el9.noarch 2024-05-06T03:37:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-06T03:37:26.205 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:26.205 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:26.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-05-06T03:37:27.049 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:34 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:27.174 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:27.174 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 127 k 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:27.177 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 127 k 2024-05-06T03:37:27.178 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 456 k 2024-05-06T03:37:27.178 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:27.404 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-19.0.0-3516.ge08cbbf6.el9.x86_64 557 kB/s | 127 kB 00:00 2024-05-06T03:37:27.404 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:27.405 INFO:teuthology.orchestra.run.smithi163.stdout:Total 552 kB/s | 127 kB 00:00 2024-05-06T03:37:27.405 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:27.410 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:27.410 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:27.472 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:27.473 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:27.623 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:27.734 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:28.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:28.382 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:28.383 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:28.383 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:37:28.383 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:28.383 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:28.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-05-06T03:37:29.114 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:36 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:29.198 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:29.241 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:29.242 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:29.242 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:29.321 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-05-06T03:37:29.838 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:36 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:29.957 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 32 k 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:37:29.958 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:29.959 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 32 k 2024-05-06T03:37:29.959 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 142 k 2024-05-06T03:37:29.959 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:30.125 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-19.0.0-3516.ge08cbbf6.el9.x86_6 191 kB/s | 32 kB 00:00 2024-05-06T03:37:30.126 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:30.126 INFO:teuthology.orchestra.run.smithi163.stdout:Total 189 kB/s | 32 kB 00:00 2024-05-06T03:37:30.126 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:30.129 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:30.129 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:30.132 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:30.132 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:30.219 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:30.320 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:30.655 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:30.865 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:30.865 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:30.865 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:30.865 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:37:30.865 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:30.865 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:31.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-05-06T03:37:31.558 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:38 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:31.638 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:31.679 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:31.679 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:31.680 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:31.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-05-06T03:37:32.263 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:39 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:32.342 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:32.383 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:32.384 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:32.384 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:32.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-05-06T03:37:32.972 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:39 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:33.057 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:33.098 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:33.099 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:33.099 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:33.165 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-05-06T03:37:33.672 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:40 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:33.751 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:33.792 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:33.793 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:33.793 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:33.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-05-06T03:37:34.376 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:41 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:34.455 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:34.496 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:34.497 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:34.497 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:34.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-05-06T03:37:35.087 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:35.167 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 is already installed. 2024-05-06T03:37:35.207 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:35.208 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-05-06T03:37:35.208 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:35.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-05-06T03:37:35.804 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:35.923 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:35.924 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:35.924 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-05-06T03:37:35.924 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:35.924 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:35.924 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 85 k 2024-05-06T03:37:35.924 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 85 k 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 231 k 2024-05-06T03:37:35.925 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:36.429 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-19.0.0-3516.ge08cbbf6.el9.x86_64.rpm 169 kB/s | 85 kB 00:00 2024-05-06T03:37:36.430 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:36.430 INFO:teuthology.orchestra.run.smithi163.stdout:Total 169 kB/s | 85 kB 00:00 2024-05-06T03:37:36.430 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:36.437 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:36.437 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:36.503 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:36.503 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:36.642 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:36.734 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:37.053 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:37.281 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:37.282 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:37.282 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:37.282 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:37:37.282 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:37.282 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:37.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-05-06T03:37:38.071 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:45 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:38.201 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:38.202 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:38.202 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-05-06T03:37:38.202 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 3.1 M 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.1 M 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 13 M 2024-05-06T03:37:38.203 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:39.492 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-19.0.0-3516.ge08cbbf6.el9.x86_64.rpm 2.4 MB/s | 3.1 MB 00:01 2024-05-06T03:37:39.493 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:39.493 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.4 MB/s | 3.1 MB 00:01 2024-05-06T03:37:39.494 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:39.503 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:39.503 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:39.584 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:39.584 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:40.010 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:40.085 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:40.110 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:40.111 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-06T03:37:40.111 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-06T03:37:40.111 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-06T03:37:40.111 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-06T03:37:40.111 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:41.105 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:41.105 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:41.105 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:41.105 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:37:41.105 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:41.105 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:41.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-05-06T03:37:41.870 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:48 ago on Mon 06 May 2024 03:35:53 AM UTC. 2024-05-06T03:37:41.993 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-3516.ge08cbbf6.el9 ceph 170 k 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-06T03:37:41.994 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-05-06T03:37:41.995 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-05-06T03:37:41.995 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:41.995 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 170 k 2024-05-06T03:37:41.995 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 494 k 2024-05-06T03:37:41.995 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-06T03:37:42.222 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-19.0.0-3516.ge08cbbf6.el9.x86_64.rpm 754 kB/s | 170 kB 00:00 2024-05-06T03:37:42.222 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-06T03:37:42.222 INFO:teuthology.orchestra.run.smithi163.stdout:Total 749 kB/s | 170 kB 00:00 2024-05-06T03:37:42.222 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-06T03:37:42.230 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-06T03:37:42.230 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-06T03:37:42.295 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-06T03:37:42.295 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-06T03:37:42.452 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-06T03:37:42.551 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:42.988 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 1/1 2024-05-06T03:37:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:43.254 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-06T03:37:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-3516.ge08cbbf6.el9.x86_64 2024-05-06T03:37:43.255 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-06T03:37:43.255 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-06T03:37:43.460 DEBUG:teuthology.parallel:result is None 2024-05-06T03:37:43.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 2024-05-06T03:37:43.689 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-06T03:37:43.710 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-3516.ge08cbbf6.el9 2024-05-06T03:37:43.710 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3516.ge08cbbf6.el9 2024-05-06T03:37:43.710 INFO:teuthology.task.install:The correct ceph version 19.0.0-3516.ge08cbbf6 is installed. 2024-05-06T03:37:43.712 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-06T03:37:43.712 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:37:43.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-06T03:37:43.774 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-06T03:37:43.774 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:37:43.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-05-06T03:37:43.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-06T03:37:43.901 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-06T03:37:43.901 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:37:43.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-06T03:37:43.964 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-06T03:37:44.027 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-06T03:37:44.028 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:37:44.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-05-06T03:37:44.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-06T03:37:44.153 INFO:teuthology.run_tasks:Running task workunit... 2024-05-06T03:37:44.167 INFO:tasks.workunit:Pulling workunits from ref e08cbbf6fd79a8dd2949f340bce25e28ea3db004 2024-05-06T03:37:44.167 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-06T03:37:44.168 INFO:tasks.workunit:timeout=3h 2024-05-06T03:37:44.168 INFO:tasks.workunit:cleanup=True 2024-05-06T03:37:44.168 DEBUG:teuthology.orchestra.run.smithi163:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-06T03:37:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T03:37:44.208 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-06T03:37:44.208 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-06T03:37:44.263 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-06T03:37:44.263 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-06T03:37:44.319 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e08cbbf6fd79a8dd2949f340bce25e28ea3db004 2024-05-06T03:37:44.376 INFO:tasks.workunit.client.0.smithi163.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr:Note: switching to 'e08cbbf6fd79a8dd2949f340bce25e28ea3db004'. 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr:state without impacting any branches by switching back to a branch. 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.961 INFO:tasks.workunit.client.0.smithi163.stderr: git switch -c 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr:Or undo this operation with: 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr: git switch - 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-05-06T03:38:28.962 INFO:tasks.workunit.client.0.smithi163.stderr:HEAD is now at e08cbbf6fd7 Merge PR #54968 into wip-yuriw-testing-20240503.181540-main 2024-05-06T03:38:28.965 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-06T03:38:29.021 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-06T03:38:29.021 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-06T03:38:29.077 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-05-06T03:38:29.077 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-05-06T03:38:29.078 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-05-06T03:38:29.150 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/mclock-config 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-05-06T03:38:29.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-05-06T03:38:29.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-05-06T03:38:29.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-05-06T03:38:29.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-05-06T03:38:29.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-05-06T03:38:29.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none ' 2024-05-06T03:38:29.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-05-06T03:38:29.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-05-06T03:38:29.158 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-05-06T03:38:29.158 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-06T03:38:29.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-05-06T03:38:29.160 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_profile_builtin_to_custom 2024-05-06T03:38:29.160 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_profile_custom_to_builtin 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_profile_disallow_builtin_params_modify 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_profile_disallow_builtin_params_override 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_recovery_limit_adjustment_mclock' 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:38:29.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:38:29.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:38:29.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:38:29.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:38:29.162 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:38:29.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:38:29.163 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:38:29.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:38:29.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:38:29.164 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:38:29.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:38:29.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:38:29.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:38:29.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:38:29.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:38:29.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:38:29.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:38:29.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:38:29.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:29.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:29.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:38:29.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:38:29.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:38:29.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:38:29.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:38:29.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:29.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:29.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-05-06T03:38:29.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:38:29.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:38:29.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:38:29.174 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:38:29.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:38:29.175 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:38:29.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:38:29.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:38:29.175 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:38:29.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:38:29.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:38:29.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:38:29.176 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:38:29.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:38:29.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:38:29.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:38:29.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:38:29.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:29.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:29.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:38:29.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:38:29.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:38:29.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:38:29.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:38:29.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:29.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:29.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-05-06T03:38:29.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:38:29.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:38:29.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:38:29.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-05-06T03:38:29.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:38:29.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:38:29.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:38:29.928 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:29.929 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:29.929 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:38:29.929 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:38:29.929 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get fsid 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:38:30.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:38:30.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get mon_host 2024-05-06T03:38:30.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-05-06T03:38:30.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-05-06T03:38:30.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:38:30.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:38:30.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:38:30.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-05-06T03:38:30.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:38:30.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:38:30.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:38:30.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:38:30.940 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:38:30.940 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:30.940 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:30.940 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:38:30.940 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:38:30.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:38:30.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:38:30.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-05-06T03:38:30.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:38:30.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e17d202a-d50d-4407-8b4d-362abb88ca9e 2024-05-06T03:38:30.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e17d202a-d50d-4407-8b4d-362abb88ca9e' 2024-05-06T03:38:30.971 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 e17d202a-d50d-4407-8b4d-362abb88ca9e 2024-05-06T03:38:30.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:38:30.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2UDhm7rROOhAAz9AfIW9sMBB94sLo5JjcfQ== 2024-05-06T03:38:30.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2UDhm7rROOhAAz9AfIW9sMBB94sLo5JjcfQ=="}' 2024-05-06T03:38:30.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e17d202a-d50d-4407-8b4d-362abb88ca9e -i td/mclock-config/0/new.json 2024-05-06T03:38:31.270 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:38:31.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-05-06T03:38:31.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQC2UDhm7rROOhAAz9AfIW9sMBB94sLo5JjcfQ== --osd-uuid e17d202a-d50d-4407-8b4d-362abb88ca9e 2024-05-06T03:38:31.314 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.308+0000 7fc4daa63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:38:31.347 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.340+0000 7fc4daa63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:38:31.348 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.341+0000 7fc4daa63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:38:31.348 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.341+0000 7fc4daa63640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:38:31.348 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.341+0000 7fc4daa63640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:38:31.348 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.341+0000 7fc4daa63640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:38:31.389 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:31.382+0000 7fc4daa63640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-05-06T03:38:37.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-05-06T03:38:37.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:38:37.234 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:38:37.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:38:37.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:38:37.784 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:38:37.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:38:37.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T03:38:37.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:38:37.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:38:37.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:38:37.809 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:37.802+0000 7f73bb187640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:38:37.811 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:37.804+0000 7f73bb187640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:38:37.812 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:37.805+0000 7f73bb187640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:38.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:38:38.210 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:38:38.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:38.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:38.334 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:38.326+0000 7f73bb187640 -1 Falling back to public interface 2024-05-06T03:38:38.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:39.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:39.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:39.604 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:38:39.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:38:39.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:39.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:40.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:40.625 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:38:40.618+0000 7f73bb187640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:38:41.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:41.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:41.038 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:38:41.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:38:41.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:41.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:41.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:42.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:42.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:42.613 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:38:42.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:38:42.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:42.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:43.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:44.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:44.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:44.052 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:38:44.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:38:44.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:44.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:45.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:45.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:45.503 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:38:45.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:38:45.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:45.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:45.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:46.923 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:38:46.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:38:46.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:46.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:47.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:48.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:48.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:48.348 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:38:48.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:38:48.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:48.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:48.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:49.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:49.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:49.762 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:38:49.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:38:49.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:49.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:50.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:51.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:51.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:51.168 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:38:51.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:38:51.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:51.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:51.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:52.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:52.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:52.773 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:38:52.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:38:52.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:52.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:53.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:54.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:54.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:54.271 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:38:54.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:38:54.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:54.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:54.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:55.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:55.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:55.671 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:38:55.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:38:55.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:55.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:56.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:57.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:57.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:57.240 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:38:57.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:38:57.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:57.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:58.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:38:59.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:38:59.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:38:59.011 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:38:59.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:38:59.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:38:59.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:38:59.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:00.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:00.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:00.602 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:39:00.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:39:00.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:00.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:01.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:02.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:02.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:02.167 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:39:02.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:39:02.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:02.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:03.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:04.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:04.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:04.561 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:39:04.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:39:04.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:04.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:05.021 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3619017406,v1:127.0.0.1:6803/3619017406] [v2:127.0.0.1:6804/3619017406,v1:127.0.0.1:6805/3619017406] exists,up e17d202a-d50d-4407-8b4d-362abb88ca9e 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:05.023 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:05.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:39:05.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-05-06T03:39:05.108 INFO:tasks.workunit.client.0.smithi163.stdout:osd_max_backfills: 1 2024-05-06T03:39:05.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-05-06T03:39:05.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-05-06T03:39:05.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-05-06T03:39:07.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-05-06T03:39:07.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:07.807 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:07.807 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:07.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:39:07.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:07.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:07.908 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:07.908 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok dump_recovery_reservations 2024-05-06T03:39:07.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-05-06T03:39:07.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-05-06T03:39:07.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-05-06T03:39:07.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:07.995 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:07.996 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok dump_recovery_reservations 2024-05-06T03:39:08.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-05-06T03:39:08.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-05-06T03:39:08.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-05-06T03:39:08.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-05-06T03:39:08.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-05-06T03:39:10.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-05-06T03:39:10.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:10.972 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:10.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:39:11.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-05-06T03:39:11.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-05-06T03:39:11.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-05-06T03:39:11.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:11.069 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:11.069 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:11.069 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:11.070 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:11.070 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:11.070 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:11.070 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:11.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:11.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok dump_recovery_reservations 2024-05-06T03:39:11.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-05-06T03:39:11.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-05-06T03:39:11.159 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-05-06T03:39:11.159 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:11.160 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:11.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:11.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok dump_recovery_reservations 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:11.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:11.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:11.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-05-06T03:39:11.972 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already down. 2024-05-06T03:39:11.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-05-06T03:39:11.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-05-06T03:39:11.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:39:11.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:39:11.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:39:11.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:11.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:39:11.986 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:39:11.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:11.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-05-06T03:39:12.426 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/3619017406,v1:127.0.0.1:6803/3619017406] [v2:127.0.0.1:6804/3619017406,v1:127.0.0.1:6805/3619017406] exists e17d202a-d50d-4407-8b4d-362abb88ca9e 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T03:39:12.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:12.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:39:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T03:39:12.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-05-06T03:39:12.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-05-06T03:39:12.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T03:39:12.433 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:39:12.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-05-06T03:39:12.434 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-05-06T03:39:12.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T03:39:12.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T03:39:12.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T03:39:12.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T03:39:12.454 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:12.447+0000 7fd86a80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:12.455 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:12.449+0000 7fd86a80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:12.456 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:12.450+0000 7fd86a80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T03:39:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:39:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:39:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:39:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:39:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:12.846 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:39:12.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:39:12.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:12.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:12.979 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:12.972+0000 7fd86a80c640 -1 Falling back to public interface 2024-05-06T03:39:13.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:14.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:14.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:14.257 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:39:14.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:39:14.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:14.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:14.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:15.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:15.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:15.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:39:15.699 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:39:15.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:15.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:15.723 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:15.716+0000 7fd86a80c640 -1 osd.0 6 log_to_monitors true 2024-05-06T03:39:16.136 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/4134450012,v1:127.0.0.1:6803/4134450012] [v2:127.0.0.1:6804/4134450012,v1:127.0.0.1:6805/4134450012] exists,up e17d202a-d50d-4407-8b4d-362abb88ca9e 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:16.138 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:16.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:39:16.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-05-06T03:39:16.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-05-06T03:39:16.224 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-05-06T03:39:16.225 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:16.225 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:16.225 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:16.225 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:16.225 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:16.226 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.226 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.226 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:16.226 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:16.226 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok dump_recovery_reservations 2024-05-06T03:39:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-05-06T03:39:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:39:16.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok dump_recovery_reservations 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:16.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:16.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:16.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:16.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:16.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:16.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:16.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:16.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:16.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:16.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:16.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:16.516 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:16.516 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.516 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:16.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:16.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:16.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:16.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:16.519 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:16.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:16.520 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:16.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:16.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:16.521 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:16.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:16.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:16.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:16.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:16.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:16.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:16.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:16.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:16.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:16.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:16.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:16.528 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:16.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:16.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:16.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:16.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:16.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:16.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:16.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:16.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:16.531 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:16.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:16.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:16.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:16.534 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:16.534 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.534 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:16.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:16.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:16.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:39:16.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:39:16.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-05-06T03:39:16.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-05-06T03:39:16.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:39:16.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:39:16.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:39:16.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-05-06T03:39:16.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:16.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:39:17.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:39:17.161 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:39:17.162 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:17.162 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:17.162 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:17.162 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:39:17.162 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:39:17.162 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get fsid 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:17.243 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:17.244 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:39:17.244 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:39:17.244 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get mon_host 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-05-06T03:39:17.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:18.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:39:18.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:39:18.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:39:18.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:39:18.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f22a720-2f5f-4f50-839d-f269acd43a86 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0f22a720-2f5f-4f50-839d-f269acd43a86' 2024-05-06T03:39:18.197 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 0f22a720-2f5f-4f50-839d-f269acd43a86 2024-05-06T03:39:18.198 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:39:18.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmUDhmSwksDBAAt3uUga4/IGbTe9c9O+4NlA== 2024-05-06T03:39:18.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmUDhmSwksDBAAt3uUga4/IGbTe9c9O+4NlA=="}' 2024-05-06T03:39:18.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f22a720-2f5f-4f50-839d-f269acd43a86 -i td/mclock-config/0/new.json 2024-05-06T03:39:18.476 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:39:18.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-05-06T03:39:18.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDmUDhmSwksDBAAt3uUga4/IGbTe9c9O+4NlA== --osd-uuid 0f22a720-2f5f-4f50-839d-f269acd43a86 2024-05-06T03:39:18.514 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.506+0000 7f8536d66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:18.544 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.537+0000 7f8536d66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:18.546 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.539+0000 7f8536d66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:18.546 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.539+0000 7f8536d66640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:39:18.547 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.539+0000 7f8536d66640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:39:18.547 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.539+0000 7f8536d66640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:39:18.584 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:18.577+0000 7f8536d66640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-05-06T03:39:24.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-05-06T03:39:24.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:39:24.003 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:39:24.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:39:24.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:39:24.561 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:39:24.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:39:24.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T03:39:24.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:39:24.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:39:24.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:39:24.587 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:24.580+0000 7fa610430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:24.589 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:24.582+0000 7fa610430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:24.590 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:24.583+0000 7fa610430640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:25.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:39:25.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:39:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:39:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:39:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:39:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:25.034 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:39:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:39:25.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:25.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:25.113 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:25.106+0000 7fa610430640 -1 Falling back to public interface 2024-05-06T03:39:25.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:26.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:26.469 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:26.469 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:39:26.469 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:39:26.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:26.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:26.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:27.463 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:27.456+0000 7fa610430640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:39:27.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:27.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:27.903 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:39:27.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:39:27.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:27.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:28.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:29.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:29.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:29.510 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:39:29.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:39:29.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:29.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:29.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:30.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:30.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:30.926 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:39:30.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:39:30.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:30.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:31.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:32.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:32.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:32.333 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:39:32.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:39:32.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:32.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:32.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:33.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:33.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:33.750 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:39:33.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:39:33.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:33.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:34.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:35.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:35.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:35.282 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:39:35.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:39:35.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:35.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:35.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:36.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:36.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:36.699 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:39:36.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:39:36.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:36.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:37.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:38.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:38.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:38.165 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:39:38.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:39:38.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:38.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:38.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:39.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:39.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:39.734 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:39:39.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:39:39.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:39.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:40.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:41.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:41.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:41.212 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:39:41.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:39:41.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:41.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:42.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:43.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:43.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:43.014 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:39:43.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:39:43.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:43.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:43.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:44.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:44.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:44.737 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:39:44.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:39:44.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:44.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:45.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:46.979 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:39:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:39:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:47.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:39:48.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:39:48.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:48.922 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:39:48.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:39:48.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:48.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:49.351 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2292490912,v1:127.0.0.1:6803/2292490912] [v2:127.0.0.1:6804/2292490912,v1:127.0.0.1:6805/2292490912] exists,up 0f22a720-2f5f-4f50-839d-f269acd43a86 2024-05-06T03:39:49.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:39:49.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:39:49.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:39:49.352 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-05-06T03:39:49.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-05-06T03:39:49.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-05-06T03:39:49.748 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-05-06T03:39:49.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:39:49.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-05-06T03:39:49.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-05-06T03:39:49.835 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-05-06T03:39:49.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-05-06T03:39:49.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-05-06T03:39:49.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-05-06T03:39:49.968 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:39:49.968 INFO:tasks.workunit.client.0.smithi163.stdout: "success": "" 2024-05-06T03:39:49.968 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:49.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-05-06T03:39:49.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:39:50.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-05-06T03:39:50.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-05-06T03:39:50.064 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-05-06T03:39:50.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-05-06T03:39:50.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-05-06T03:39:50.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-05-06T03:39:50.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-05-06T03:39:50.066 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-05-06T03:39:50.066 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:50.066 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:50.067 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:50.067 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:50.067 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:50.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:50.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-05-06T03:39:50.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:39:50.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-05-06T03:39:50.151 INFO:tasks.workunit.client.0.smithi163.stdout:client_res = 0.300000 2024-05-06T03:39:50.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-05-06T03:39:50.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-05-06T03:39:50.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-05-06T03:39:50.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-05-06T03:39:50.152 INFO:tasks.workunit.client.0.smithi163.stdout:client_res_new = .400000 2024-05-06T03:39:50.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-05-06T03:39:50.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-05-06T03:39:50.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:39:51.046 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-05-06T03:39:51.047 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-05-06T03:39:51.047 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-05-06T03:39:51.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-05-06T03:39:51.049 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-05-06T03:39:51.049 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-05-06T03:39:51.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-05-06T03:39:51.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:39:51.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:39:51.050 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:51.050 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.050 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.050 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:39:51.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-05-06T03:39:51.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:39:51.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-05-06T03:39:51.134 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-05-06T03:39:51.135 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-05-06T03:39:51.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-05-06T03:39:51.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-05-06T03:39:51.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:51.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:51.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:51.137 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:51.137 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:51.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:51.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:51.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:51.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:51.242 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:51.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:51.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:51.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:51.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:51.244 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:51.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:51.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:51.245 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:51.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:51.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:51.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:51.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:51.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:51.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:51.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:51.255 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:51.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:51.256 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:51.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:51.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:51.257 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:51.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:51.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:51.258 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:51.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:51.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:51.261 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:51.261 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.261 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:51.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:51.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:51.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:51.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:51.264 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:51.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:51.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:51.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:51.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:51.266 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:51.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:51.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:51.267 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:51.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:51.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:51.270 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:51.270 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.270 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:51.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:51.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:51.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:39:51.272 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:39:51.272 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.272 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:39:51.274 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:39:51.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:39:51.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:39:51.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:39:51.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:39:51.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:39:51.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:39:51.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:39:51.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:39:51.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:39:51.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:39:51.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:39:51.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:39:51.279 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:39:51.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:39:51.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:39:51.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:39:51.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:39:51.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:39:51.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:39:51.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:39:51.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:39:51.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:39:51.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-05-06T03:39:51.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-05-06T03:39:51.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:39:51.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:51.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:51.699 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:51.700 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.700 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:51.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:39:51.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:39:51.885 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:51.886 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.886 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.886 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:39:51.886 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:39:51.886 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get fsid 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:39:51.966 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:39:51.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get mon_host 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-05-06T03:39:52.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:39:52.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:39:52.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:52.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:52.889 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:52.890 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:52.890 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:52.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:52.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:39:52.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:39:52.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-05-06T03:39:52.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-05-06T03:39:52.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:39:52.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:39:52.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:39:52.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-05-06T03:39:52.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:39:52.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:39:52.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:39:52.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:39:52.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:39:52.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:39:52.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:39:52.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:39:52.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:39:52.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:39:52.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:39:52.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:39:52.924 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:39:52.924 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:39:52.924 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:39:52.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:52.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:39:52.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:39:52.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:39:52.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:39:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:39:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:39:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:39:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:39:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:39:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:39:52.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:39:52.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T03:39:52.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-05-06T03:39:52.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:39:52.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a83c702-1199-422c-a961-6ae0076d1343 2024-05-06T03:39:52.928 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 9a83c702-1199-422c-a961-6ae0076d1343 2024-05-06T03:39:52.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9a83c702-1199-422c-a961-6ae0076d1343' 2024-05-06T03:39:52.929 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:39:52.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIUThm13BaNxAAkSBSN0jsFT3SQ0GQkFf4jQ== 2024-05-06T03:39:52.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIUThm13BaNxAAkSBSN0jsFT3SQ0GQkFf4jQ=="}' 2024-05-06T03:39:52.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a83c702-1199-422c-a961-6ae0076d1343 -i td/mclock-config/0/new.json 2024-05-06T03:39:53.216 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:39:53.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-05-06T03:39:53.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAIUThm13BaNxAAkSBSN0jsFT3SQ0GQkFf4jQ== --osd-uuid 9a83c702-1199-422c-a961-6ae0076d1343 2024-05-06T03:39:53.254 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.246+0000 7ff0d7a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:53.284 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.277+0000 7ff0d7a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:53.285 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.278+0000 7ff0d7a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:53.286 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.278+0000 7ff0d7a11640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:39:53.286 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.279+0000 7ff0d7a11640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:39:53.286 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.279+0000 7ff0d7a11640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:39:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:53.317+0000 7ff0d7a11640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-05-06T03:39:58.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-05-06T03:39:58.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:39:58.797 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:39:58.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:39:58.798 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:39:59.366 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:39:59.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:39:59.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T03:39:59.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:39:59.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:39:59.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:39:59.392 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:59.384+0000 7febf0e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:59.394 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:59.387+0000 7febf0e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:59.395 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:59.388+0000 7febf0e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:39:59.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:39:59.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:39:59.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:39:59.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:39:59.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:39:59.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:39:59.794 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:39:59.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:39:59.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:39:59.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:39:59.909 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:39:59.902+0000 7febf0e1e640 -1 Falling back to public interface 2024-05-06T03:40:00.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:01.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:01.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:01.187 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:40:01.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:40:01.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:01.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:01.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:02.265 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:02.257+0000 7febf0e1e640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:40:02.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:02.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:02.621 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:40:02.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:40:02.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:02.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:03.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:04.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:04.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:04.059 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:40:04.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:40:04.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:04.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:04.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:05.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:05.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:05.462 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:40:05.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:40:05.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:05.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:05.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:06.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:06.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:06.861 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:40:06.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:40:06.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:06.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:07.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:08.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:08.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:08.422 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:40:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:40:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:08.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:09.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:09.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:09.857 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:40:09.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:40:09.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:09.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:10.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:11.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:11.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:11.300 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:40:11.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:40:11.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:11.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:11.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:12.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:12.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:12.701 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:40:12.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:40:12.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:12.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:13.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:14.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:14.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:14.218 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:40:14.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:40:14.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:14.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:14.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:15.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:15.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:15.843 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:40:15.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:40:15.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:15.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:16.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:17.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:17.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:17.540 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:40:17.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:40:17.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:17.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:18.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:19.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:19.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:19.283 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:40:19.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:40:19.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:19.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:20.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:21.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:21.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:21.054 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:40:21.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:40:21.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:21.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:21.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:22.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:22.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:22.571 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:40:22.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:40:22.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:22.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:23.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:24.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:24.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:24.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:40:24.756 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:40:24.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:24.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:25.300 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3541239705,v1:127.0.0.1:6803/3541239705] [v2:127.0.0.1:6804/3541239705,v1:127.0.0.1:6805/3541239705] exists,up 9a83c702-1199-422c-a961-6ae0076d1343 2024-05-06T03:40:25.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:40:25.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:40:25.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:40:25.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-05-06T03:40:25.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-05-06T03:40:25.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-05-06T03:40:25.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:25.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:25.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:40:25.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-05-06T03:40:25.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-05-06T03:40:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-05-06T03:40:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-05-06T03:40:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-05-06T03:40:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-05-06T03:40:25.886 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:40:25.887 INFO:tasks.workunit.client.0.smithi163.stdout: "success": "" 2024-05-06T03:40:25.887 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:25.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:40:25.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-05-06T03:40:25.981 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-05-06T03:40:25.982 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-05-06T03:40:25.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-05-06T03:40:25.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-05-06T03:40:25.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-05-06T03:40:25.983 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:25.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:40:26.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-05-06T03:40:26.068 INFO:tasks.workunit.client.0.smithi163.stdout:Original client_res for osd.0 = 0.300000 2024-05-06T03:40:26.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-05-06T03:40:26.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-05-06T03:40:26.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:26.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-05-06T03:40:26.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-05-06T03:40:26.069 INFO:tasks.workunit.client.0.smithi163.stdout:client_res_new = .400000 2024-05-06T03:40:26.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-05-06T03:40:26.528 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:40:26.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-05-06T03:40:26.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-05-06T03:40:26.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:26.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-05-06T03:40:26.957 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-05-06T03:40:26.957 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:26.958 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:26.959 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:40:27.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-05-06T03:40:27.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-05-06T03:40:27.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-05-06T03:40:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-05-06T03:40:27.170 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:40:27.170 INFO:tasks.workunit.client.0.smithi163.stdout: "success": "" 2024-05-06T03:40:27.170 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:40:27.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-05-06T03:40:27.180 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:27.180 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:27.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:40:27.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-05-06T03:40:27.266 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-05-06T03:40:27.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-05-06T03:40:27.267 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:40:27.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-05-06T03:40:27.661 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-05-06T03:40:27.661 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:27.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-05-06T03:40:27.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-05-06T03:40:27.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:27.664 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:40:27.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-05-06T03:40:27.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-05-06T03:40:27.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:27.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-05-06T03:40:27.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-05-06T03:40:28.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-05-06T03:40:33.248 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:40:33.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-05-06T03:40:33.685 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-05-06T03:40:33.685 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:33.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-05-06T03:40:33.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:40:33.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-05-06T03:40:33.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-05-06T03:40:33.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-05-06T03:40:33.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-05-06T03:40:33.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:40:33.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:40:33.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:40:33.881 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:40:33.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:40:33.882 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:40:33.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:40:33.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:40:33.883 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:40:33.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:40:33.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:40:33.884 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:40:33.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:40:33.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:40:33.891 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:40:33.891 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.891 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:40:33.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:40:33.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:40:33.894 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:40:33.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:40:33.895 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:40:33.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:40:33.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:40:33.896 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:40:33.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:40:33.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:40:33.898 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:40:33.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:40:33.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:40:33.901 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:40:33.901 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.901 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:40:33.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:40:33.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:40:33.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:40:33.904 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:40:33.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:40:33.904 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:40:33.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:40:33.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:40:33.905 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:40:33.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:40:33.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:40:33.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:40:33.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:40:33.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:40:33.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:40:33.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:40:33.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:40:33.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:40:33.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:40:33.911 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:40:33.911 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.911 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:40:33.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:40:33.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:40:33.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:40:33.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:40:33.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:40:33.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:40:33.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:40:33.916 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:40:33.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:40:33.916 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:40:33.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:40:33.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:40:33.917 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:40:33.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:40:33.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:40:33.918 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:40:33.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:40:33.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:40:33.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:40:33.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:40:33.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:40:33.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:40:33.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:40:33.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:40:33.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:40:33.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:33.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:33.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:40:33.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:40:33.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-05-06T03:40:33.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-05-06T03:40:34.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:40:34.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:40:34.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:40:34.381 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:40:34.381 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:34.381 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:34.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:40:34.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:40:34.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:40:34.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:40:34.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get fsid 2024-05-06T03:40:34.646 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:40:34.646 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:40:34.646 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:40:34.647 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get mon_host 2024-05-06T03:40:34.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-05-06T03:40:34.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-05-06T03:40:34.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:40:34.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:40:34.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:40:34.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-05-06T03:40:34.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:40:35.577 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:40:35.577 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:40:35.577 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:40:35.578 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:40:35.578 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:35.578 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:35.578 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:40:35.578 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:40:35.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:40:35.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-05-06T03:40:35.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-05-06T03:40:35.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:40:35.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:40:35.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:40:35.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:40:35.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc21d178-d19a-444e-8274-712a14395b5c 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cc21d178-d19a-444e-8274-712a14395b5c' 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 cc21d178-d19a-444e-8274-712a14395b5c 2024-05-06T03:40:35.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:40:35.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzUThmG6a6JBAAQDcdpnPY+Se2FwzJDQpqGA== 2024-05-06T03:40:35.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzUThmG6a6JBAAQDcdpnPY+Se2FwzJDQpqGA=="}' 2024-05-06T03:40:35.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc21d178-d19a-444e-8274-712a14395b5c -i td/mclock-config/0/new.json 2024-05-06T03:40:35.904 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:40:35.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-05-06T03:40:35.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAzUThmG6a6JBAAQDcdpnPY+Se2FwzJDQpqGA== --osd-uuid cc21d178-d19a-444e-8274-712a14395b5c 2024-05-06T03:40:35.942 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.935+0000 7f7052e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:40:35.973 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.965+0000 7f7052e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:40:35.974 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.967+0000 7f7052e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:40:35.974 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.967+0000 7f7052e5f640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:40:35.975 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.967+0000 7f7052e5f640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:40:35.975 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.967+0000 7f7052e5f640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:40:36.005 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:35.997+0000 7f7052e5f640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-05-06T03:40:41.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-05-06T03:40:41.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:40:41.504 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:40:41.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:40:41.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:40:42.072 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:40:42.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:40:42.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T03:40:42.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:40:42.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:40:42.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:40:42.098 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:42.090+0000 7fc45b010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:40:42.109 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:42.102+0000 7fc45b010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:40:42.110 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:42.103+0000 7fc45b010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:40:42.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:40:42.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:40:42.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:40:42.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:40:42.540 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:40:42.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:40:42.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:42.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:40:42.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:42.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:42.652 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:42.645+0000 7fc45b010640 -1 Falling back to public interface 2024-05-06T03:40:42.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:43.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:43.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:43.967 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:40:43.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:40:43.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:43.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:44.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:45.011 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:40:45.003+0000 7fc45b010640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:40:45.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:45.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:45.401 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:40:45.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:40:45.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:45.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:45.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:46.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:46.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:46.933 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:40:46.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:40:46.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:46.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:47.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:48.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:48.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:48.334 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:40:48.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:40:48.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:48.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:48.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:49.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:49.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:49.743 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:40:49.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:40:49.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:49.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:50.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:51.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:51.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:51.172 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:40:51.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:40:51.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:51.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:51.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:52.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:52.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:52.578 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:40:52.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:40:52.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:52.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:53.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:54.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:54.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:54.074 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:40:54.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:40:54.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:54.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:54.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:55.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:55.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:55.497 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:40:55.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:40:55.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:55.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:55.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:56.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:56.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:56.907 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:40:56.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:40:56.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:56.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:57.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:40:58.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:40:58.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:40:58.573 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:40:58.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:40:58.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:40:58.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:40:59.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:00.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:00.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:00.185 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:41:00.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:41:00.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:00.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:01.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:02.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:02.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:02.271 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:41:02.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:41:02.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:02.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:03.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:04.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:04.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:04.397 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:41:04.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:41:04.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:04.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:05.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:06.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:06.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:06.278 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:41:06.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:41:06.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:06.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:07.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:08.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:08.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:08.196 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:41:08.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:41:08.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:08.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:08.620 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2802769388,v1:127.0.0.1:6803/2802769388] [v2:127.0.0.1:6804/2802769388,v1:127.0.0.1:6805/2802769388] exists,up cc21d178-d19a-444e-8274-712a14395b5c 2024-05-06T03:41:08.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:41:08.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:41:08.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:41:08.621 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-05-06T03:41:09.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-05-06T03:41:09.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-05-06T03:41:09.056 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-05-06T03:41:09.056 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-05-06T03:41:09.057 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:41:09.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-05-06T03:41:09.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-05-06T03:41:09.143 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-05-06T03:41:09.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-05-06T03:41:09.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-05-06T03:41:09.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:09.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-05-06T03:41:09.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:09.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-05-06T03:41:09.230 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-05-06T03:41:09.230 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:09.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-05-06T03:41:09.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-05-06T03:41:09.832 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-05-06T03:41:09.833 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-05-06T03:41:09.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-05-06T03:41:09.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-05-06T03:41:09.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-05-06T03:41:11.867 INFO:tasks.workunit.client.0.smithi163.stdout:Check configuration values - Attempt#: 0 2024-05-06T03:41:11.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-05-06T03:41:11.867 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:12.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-05-06T03:41:12.300 INFO:tasks.workunit.client.0.smithi163.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-05-06T03:41:12.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-05-06T03:41:12.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-05-06T03:41:12.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:12.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-05-06T03:41:12.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-05-06T03:41:12.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:12.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-05-06T03:41:12.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:12.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-05-06T03:41:12.695 INFO:tasks.workunit.client.0.smithi163.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-05-06T03:41:12.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-05-06T03:41:12.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-05-06T03:41:12.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:12.696 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-05-06T03:41:12.696 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:12.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-05-06T03:41:12.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-05-06T03:41:12.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:12.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-05-06T03:41:12.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-05-06T03:41:12.785 INFO:tasks.workunit.client.0.smithi163.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-05-06T03:41:12.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-05-06T03:41:12.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:12.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-05-06T03:41:12.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-05-06T03:41:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-05-06T03:41:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-05-06T03:41:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:12.790 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-05-06T03:41:12.790 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:41:12.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-05-06T03:41:12.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-05-06T03:41:12.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:12.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-05-06T03:41:12.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-05-06T03:41:13.444 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-05-06T03:41:13.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-05-06T03:41:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-05-06T03:41:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-05-06T03:41:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-05-06T03:41:15.447 INFO:tasks.workunit.client.0.smithi163.stdout:Check configuration values - Attempt#: 0 2024-05-06T03:41:15.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-05-06T03:41:15.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:41:15.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-05-06T03:41:15.888 INFO:tasks.workunit.client.0.smithi163.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-05-06T03:41:15.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-05-06T03:41:15.888 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-05-06T03:41:15.888 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:15.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-05-06T03:41:15.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-05-06T03:41:15.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-05-06T03:41:15.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-05-06T03:41:15.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:16.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.400000 2024-05-06T03:41:16.281 INFO:tasks.workunit.client.0.smithi163.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .400000 2024-05-06T03:41:16.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .400000' 2024-05-06T03:41:16.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.400000 == .400000' 2024-05-06T03:41:16.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:16.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.400000 != 0.300000' 2024-05-06T03:41:16.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:16.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 1 || 1 )) 2024-05-06T03:41:16.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:357: TEST_profile_disallow_builtin_params_modify: expr 0 + 1 2024-05-06T03:41:16.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:357: TEST_profile_disallow_builtin_params_modify: errors=1 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:16.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:16.287 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-05-06T03:41:16.287 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:41:16.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-05-06T03:41:16.371 INFO:tasks.workunit.client.0.smithi163.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-05-06T03:41:16.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-05-06T03:41:16.372 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-05-06T03:41:16.372 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:16.374 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-05-06T03:41:16.374 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:16.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-05-06T03:41:16.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 1 -eq 0 ']' 2024-05-06T03:41:16.375 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:373: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-05-06T03:41:16.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:373: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 9 ']' 2024-05-06T03:41:16.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-05-06T03:41:16.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-05-06T03:41:16.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-05-06T03:41:18.376 INFO:tasks.workunit.client.0.smithi163.stdout:Check configuration values - Attempt#: 1 2024-05-06T03:41:18.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 1' 2024-05-06T03:41:18.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:41:18.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-05-06T03:41:18.776 INFO:tasks.workunit.client.0.smithi163.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-05-06T03:41:18.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-05-06T03:41:18.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-05-06T03:41:18.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:18.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-05-06T03:41:18.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-05-06T03:41:18.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-05-06T03:41:18.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-05-06T03:41:18.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:19.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-05-06T03:41:19.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-05-06T03:41:19.167 INFO:tasks.workunit.client.0.smithi163.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-05-06T03:41:19.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-05-06T03:41:19.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:19.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-05-06T03:41:19.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:19.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-05-06T03:41:19.171 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-05-06T03:41:19.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:41:19.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-05-06T03:41:19.256 INFO:tasks.workunit.client.0.smithi163.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-05-06T03:41:19.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-05-06T03:41:19.257 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-05-06T03:41:19.257 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:19.258 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-05-06T03:41:19.258 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:41:19.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:41:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:41:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:41:19.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:41:19.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:41:19.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:41:19.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:41:19.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:41:19.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:41:19.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:41:19.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:41:19.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:41:19.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:41:19.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:41:19.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:41:19.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:41:19.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:41:19.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:41:19.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:41:19.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-05-06T03:41:19.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:41:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:41:19.379 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:41:19.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:41:19.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:41:19.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:41:19.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:41:19.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:41:19.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:41:19.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:41:19.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:41:19.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:41:19.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:41:19.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:41:19.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:41:19.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:41:19.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:41:19.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:41:19.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:41:19.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:41:19.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:41:19.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:41:19.389 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:41:19.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:41:19.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:41:19.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:41:19.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:41:19.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:41:19.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:41:19.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:41:19.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:41:19.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:41:19.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.394 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:41:19.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:41:19.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:41:19.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:41:19.395 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:41:19.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:41:19.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:41:19.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:41:19.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:41:19.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:41:19.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:41:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:41:19.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:41:19.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:41:19.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:41:19.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:41:19.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:41:19.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:41:19.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:41:19.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:41:19.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:41:19.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:41:19.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:41:19.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:41:19.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:41:19.405 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:41:19.405 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.405 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:41:19.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:41:19.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:41:19.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:41:19.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:41:19.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:41:19.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:41:19.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:41:19.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-05-06T03:41:19.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-05-06T03:41:19.877 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:41:19.877 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:41:19.877 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:41:19.877 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:41:19.877 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:19.878 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:19.878 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:41:19.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:41:20.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:41:20.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:41:20.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:41:20.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:41:20.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:41:20.072 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:41:20.073 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get fsid 2024-05-06T03:41:20.153 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:41:20.153 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:41:20.153 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:41:20.153 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:41:20.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get mon_host 2024-05-06T03:41:20.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-05-06T03:41:20.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-05-06T03:41:20.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:41:20.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:41:20.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:41:20.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-05-06T03:41:20.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:41:21.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:41:21.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:41:21.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:41:21.075 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:41:21.075 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:21.075 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:21.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:41:21.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:41:21.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:41:21.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-05-06T03:41:21.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:41:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:41:21.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91d8964f-8a85-4a99-86ab-e334e9cee41f 2024-05-06T03:41:21.107 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 91d8964f-8a85-4a99-86ab-e334e9cee41f 2024-05-06T03:41:21.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 91d8964f-8a85-4a99-86ab-e334e9cee41f' 2024-05-06T03:41:21.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:41:21.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhUThmKmHFBhAA3GPa4+PJ2VqHU1tyS3z0hw== 2024-05-06T03:41:21.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhUThmKmHFBhAA3GPa4+PJ2VqHU1tyS3z0hw=="}' 2024-05-06T03:41:21.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91d8964f-8a85-4a99-86ab-e334e9cee41f -i td/mclock-config/0/new.json 2024-05-06T03:41:21.403 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:41:21.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-05-06T03:41:21.413 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBhUThmKmHFBhAA3GPa4+PJ2VqHU1tyS3z0hw== --osd-uuid 91d8964f-8a85-4a99-86ab-e334e9cee41f 2024-05-06T03:41:21.441 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.433+0000 7f7ef51dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:41:21.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.462+0000 7f7ef51dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:41:21.471 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.463+0000 7f7ef51dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:41:21.472 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.464+0000 7f7ef51dc640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:41:21.472 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.464+0000 7f7ef51dc640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:41:21.472 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.464+0000 7f7ef51dc640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:41:21.510 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:21.502+0000 7f7ef51dc640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-05-06T03:41:27.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-05-06T03:41:27.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:41:27.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:41:27.014 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:41:27.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:41:27.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:41:27.561 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:41:27.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T03:41:27.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:41:27.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:41:27.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:41:27.586 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:27.578+0000 7f99ca433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:41:27.599 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:27.591+0000 7f99ca433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:41:27.600 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:27.592+0000 7f99ca433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:27.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:41:27.987 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:41:27.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:27.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:28.120 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:28.112+0000 7f99ca433640 -1 Falling back to public interface 2024-05-06T03:41:28.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:29.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:29.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:29.421 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:41:29.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:41:29.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:29.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:29.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:30.457 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:41:30.448+0000 7f99ca433640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:41:30.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:30.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:30.839 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:41:30.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:41:30.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:30.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:31.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:32.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:32.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:32.237 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:41:32.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:41:32.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:32.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:32.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:33.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:33.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:33.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:41:33.657 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:41:33.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:33.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:34.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:35.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:35.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:35.059 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:41:35.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:41:35.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:35.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:35.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:36.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:36.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:36.585 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:41:36.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:41:36.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:36.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:37.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:38.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:38.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:38.016 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:41:38.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:41:38.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:38.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:38.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:39.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:39.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:39.414 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:41:39.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:41:39.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:39.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:39.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:40.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:40.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:40.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:41:40.827 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:41:40.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:40.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:41.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:42.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:42.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:42.325 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:41:42.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:41:42.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:42.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:43.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:44.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:44.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:44.229 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:41:44.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:41:44.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:44.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:44.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:45.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:45.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:45.811 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:41:45.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:41:45.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:45.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:46.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:47.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:47.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:47.561 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:41:47.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:41:47.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:47.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:48.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:49.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:49.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:49.514 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:41:49.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:41:49.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:49.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:50.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:51.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:51.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:51.333 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:41:51.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:41:51.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:51.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:52.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:41:53.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:41:53.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:41:53.227 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:41:53.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:41:53.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:41:53.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:41:53.648 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/431890540,v1:127.0.0.1:6803/431890540] [v2:127.0.0.1:6804/431890540,v1:127.0.0.1:6805/431890540] exists,up 91d8964f-8a85-4a99-86ab-e334e9cee41f 2024-05-06T03:41:53.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:41:53.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:41:53.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:41:53.649 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-05-06T03:41:54.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-05-06T03:41:54.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:54.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-05-06T03:41:54.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_profile 2024-05-06T03:41:54.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-05-06T03:41:54.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-05-06T03:41:54.154 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:54.155 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:54.156 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:54.156 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:54.156 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:54.156 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:54.156 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-05-06T03:41:54.156 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:54.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-05-06T03:41:54.240 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-05-06T03:41:54.240 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:54.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-05-06T03:41:54.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-05-06T03:41:54.344 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:41:54.344 INFO:tasks.workunit.client.0.smithi163.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-05-06T03:41:54.344 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:41:54.353 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-05-06T03:41:54.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-05-06T03:41:54.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-05-06T03:41:54.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-05-06T03:41:54.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-05-06T03:41:56.356 INFO:tasks.workunit.client.0.smithi163.stdout:Check configuration values - Attempt#: 0 2024-05-06T03:41:56.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-05-06T03:41:56.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:56.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-05-06T03:41:56.784 INFO:tasks.workunit.client.0.smithi163.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-05-06T03:41:56.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-05-06T03:41:56.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-05-06T03:41:56.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:56.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-05-06T03:41:56.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-05-06T03:41:56.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:56.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-05-06T03:41:56.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-05-06T03:41:57.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-05-06T03:41:57.183 INFO:tasks.workunit.client.0.smithi163.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-05-06T03:41:57.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-05-06T03:41:57.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-05-06T03:41:57.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:57.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-05-06T03:41:57.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:57.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-05-06T03:41:57.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-05-06T03:41:57.187 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-05-06T03:41:57.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-05-06T03:41:57.272 INFO:tasks.workunit.client.0.smithi163.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-05-06T03:41:57.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-05-06T03:41:57.273 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-05-06T03:41:57.273 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:57.273 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-05-06T03:41:57.274 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-05-06T03:41:57.276 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-05-06T03:41:57.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:41:57.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-05-06T03:41:57.362 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-05-06T03:41:57.362 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:57.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-05-06T03:41:57.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-05-06T03:41:57.466 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:41:57.466 INFO:tasks.workunit.client.0.smithi163.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-05-06T03:41:57.466 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:41:57.475 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-05-06T03:41:57.476 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-05-06T03:41:57.477 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-05-06T03:41:57.477 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-05-06T03:41:57.477 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-05-06T03:41:59.478 INFO:tasks.workunit.client.0.smithi163.stdout:Check configuration values - Attempt#: 0 2024-05-06T03:41:59.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-05-06T03:41:59.479 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-05-06T03:41:59.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-05-06T03:41:59.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-05-06T03:41:59.920 INFO:tasks.workunit.client.0.smithi163.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-05-06T03:41:59.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-05-06T03:41:59.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:41:59.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-05-06T03:41:59.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-05-06T03:41:59.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-05-06T03:41:59.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-05-06T03:41:59.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-05-06T03:42:00.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-05-06T03:42:00.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-05-06T03:42:00.310 INFO:tasks.workunit.client.0.smithi163.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-05-06T03:42:00.310 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-05-06T03:42:00.311 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:42:00.311 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-05-06T03:42:00.311 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:42:00.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-05-06T03:42:00.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-05-06T03:42:00.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-05-06T03:42:00.399 INFO:tasks.workunit.client.0.smithi163.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-05-06T03:42:00.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-05-06T03:42:00.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-05-06T03:42:00.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:42:00.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-05-06T03:42:00.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-05-06T03:42:00.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-05-06T03:42:00.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:00.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:00.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:00.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:00.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:00.510 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:00.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:00.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:00.511 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:00.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:00.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:00.512 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:00.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:00.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:00.519 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:00.519 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.519 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:00.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:00.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:00.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-05-06T03:42:00.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:00.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:00.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:00.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:00.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:00.523 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:00.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:00.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:00.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:00.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:00.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:00.526 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:00.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:00.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:00.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:00.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:00.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:00.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:00.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:00.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:00.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:00.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:00.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:00.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:00.534 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:00.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:00.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:00.535 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:00.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:00.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:00.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:00.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:00.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:00.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:00.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:42:00.539 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:42:00.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:42:00.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:42:00.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:42:00.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:00.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:00.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:00.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:00.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:00.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:00.545 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:00.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:00.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:00.546 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:00.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:00.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:00.547 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:00.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:00.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:00.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:00.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:00.550 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.550 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:00.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:00.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:00.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-05-06T03:42:00.551 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:42:00.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:00.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:00.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26179 2024-05-06T03:42:00.553 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-05-06T03:42:00.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-05-06T03:42:01.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:42:01.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:42:01.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:42:01.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:42:01.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:01.026 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:01.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:42:01.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:42:01.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:42:01.219 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:42:01.219 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:42:01.219 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:42:01.219 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:42:01.219 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:42:01.220 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get fsid 2024-05-06T03:42:01.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:42:01.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:42:01.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:42:01.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:42:01.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:42:01.302 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-mon.a.asok 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:42:01.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26179/ceph-mon.a.asok config get mon_host 2024-05-06T03:42:01.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-05-06T03:42:01.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-05-06T03:42:01.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:42:01.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:42:01.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:42:01.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-05-06T03:42:01.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:42:02.231 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:42:02.231 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:42:02.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:42:02.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-05-06T03:42:02.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-05-06T03:42:02.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:42:02.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-05-06T03:42:02.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-05-06T03:42:02.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-05-06T03:42:02.264 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:42:02.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6bad5b2d-a79e-4a49-aa05-d918a44e7811 2024-05-06T03:42:02.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6bad5b2d-a79e-4a49-aa05-d918a44e7811' 2024-05-06T03:42:02.265 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 6bad5b2d-a79e-4a49-aa05-d918a44e7811 2024-05-06T03:42:02.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:42:02.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKUThmaYwfEBAAcZiZRmg0dp1YIKsIu8/IoQ== 2024-05-06T03:42:02.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKUThmaYwfEBAAcZiZRmg0dp1YIKsIu8/IoQ=="}' 2024-05-06T03:42:02.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6bad5b2d-a79e-4a49-aa05-d918a44e7811 -i td/mclock-config/0/new.json 2024-05-06T03:42:02.558 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:42:02.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-05-06T03:42:02.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCKUThmaYwfEBAAcZiZRmg0dp1YIKsIu8/IoQ== --osd-uuid 6bad5b2d-a79e-4a49-aa05-d918a44e7811 2024-05-06T03:42:02.596 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.588+0000 7f93c9418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:02.618 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.610+0000 7f93c9418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:02.619 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.611+0000 7f93c9418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:02.620 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.612+0000 7f93c9418640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:42:02.620 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.612+0000 7f93c9418640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:42:02.620 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.612+0000 7f93c9418640 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-05-06T03:42:02.658 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:02.650+0000 7f93c9418640 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-05-06T03:42:08.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-05-06T03:42:08.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:42:08.124 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:42:08.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:42:08.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:42:08.702 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:42:08.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:42:08.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c80b0f38-74c5-4be3-9db1-35a934422891 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T03:42:08.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:42:08.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:42:08.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:42:08.727 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:08.719+0000 7fe88762c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:08.729 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:08.721+0000 7fe88762c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:08.730 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:08.722+0000 7fe88762c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:09.129 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:42:09.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:42:09.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:09.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:09.262 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:09.254+0000 7fe88762c640 -1 Falling back to public interface 2024-05-06T03:42:09.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:10.563 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:10.564 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:42:10.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:10.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:42:10.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:10.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:10.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:11.632 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:11.624+0000 7fe88762c640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:42:11.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:11.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:11.998 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:42:11.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:42:11.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:11.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:12.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:13.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:13.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:13.392 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:42:13.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:42:13.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:13.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:13.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:14.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:14.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:14.808 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:42:14.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:42:14.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:14.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:15.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:16.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:16.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:16.231 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:42:16.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:42:16.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:16.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:16.706 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:17.707 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:17.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:17.708 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:42:17.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:42:17.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:17.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:18.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:19.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:19.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:19.113 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:42:19.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:42:19.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:19.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:19.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:20.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:20.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:20.518 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:42:20.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:42:20.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:20.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:20.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:21.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:21.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:21.923 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:42:21.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:42:21.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:21.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:22.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:23.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:23.390 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:42:23.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:23.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:42:23.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:23.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:24.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:25.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:25.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:25.034 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:42:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:42:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:25.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:25.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:26.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:26.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:26.584 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:42:26.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:42:26.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:26.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:27.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:28.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:28.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:28.276 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:42:28.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:42:28.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:28.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:28.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:29.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:29.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:29.687 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:42:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:42:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:30.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:31.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:31.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:31.590 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:42:31.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:42:31.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:31.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:32.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:33.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:33.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:33.618 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:42:33.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:42:33.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:33.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:34.097 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4130400115,v1:127.0.0.1:6803/4130400115] [v2:127.0.0.1:6804/4130400115,v1:127.0.0.1:6805/4130400115] exists,up 6bad5b2d-a79e-4a49-aa05-d918a44e7811 2024-05-06T03:42:34.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:42:34.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_recovery_max_active 2024-05-06T03:42:34.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-05-06T03:42:34.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-05-06T03:42:34.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-05-06T03:42:34.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-05-06T03:42:34.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-05-06T03:42:36.866 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:42:36.867 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_recovery_max_active 2024-05-06T03:42:36.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-05-06T03:42:36.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-05-06T03:42:36.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-05-06T03:42:37.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-05-06T03:42:37.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-05-06T03:42:39.875 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26179/ceph-osd.0.asok 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-05-06T03:42:39.876 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26179/ceph-osd.0.asok config get osd_recovery_max_active 2024-05-06T03:42:39.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-05-06T03:42:39.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:39.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:40.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:40.077 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:40.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:40.078 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:40.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:40.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:40.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:40.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:40.087 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:40.087 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.087 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:40.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:40.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:40.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-05-06T03:42:40.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:40.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:40.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:40.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:40.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:40.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:40.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:40.096 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:40.096 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.096 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:40.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/mclock-config 0 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-05-06T03:42:40.098 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:40.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:40.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:40.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:40.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:40.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:40.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:40.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:40.100 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:40.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:40.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:40.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:40.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:40.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:40.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:40.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-06T03:42:40.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-05-06T03:42:40.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:40.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26179 2024-05-06T03:42:40.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26179 2024-05-06T03:42:40.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:40.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:40.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-05-06T03:42:40.107 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-06T03:42:40.107 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T03:42:40.132 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-05-06T03:42:40.132 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-05-06T03:42:40.194 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T03:42:40.199 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T03:42:40.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:42:40.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/network-ping 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-05-06T03:42:40.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none ' 2024-05-06T03:42:40.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-05-06T03:42:40.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-05-06T03:42:40.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-05-06T03:42:40.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-05-06T03:42:40.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-05-06T03:42:40.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_network_ping_test2' 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:42:40.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:42:40.204 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:42:40.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:42:40.205 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:42:40.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:42:40.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:42:40.205 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:42:40.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:42:40.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:42:40.206 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:42:40.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:42:40.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:42:40.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-05-06T03:42:40.209 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:42:40.209 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.209 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:40.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40033 2024-05-06T03:42:40.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:42:40.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:42:40.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-05-06T03:42:40.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:42:40.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:40.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.40033 2024-05-06T03:42:40.213 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:42:40.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T03:42:40.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T03:42:40.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-05-06T03:42:40.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:42:40.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:42:40.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mon.a.asok 2024-05-06T03:42:40.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:42:40.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40033/ceph-mon.a.asok config get fsid 2024-05-06T03:42:40.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:42:40.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mon.a.asok 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:42:40.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40033/ceph-mon.a.asok config get mon_host 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-05-06T03:42:41.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:41.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:42:41.861 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:42:41.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:42:41.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:42:41.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=baa3f48c-5c24-4c19-8f86-1038b8924341 2024-05-06T03:42:41.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 baa3f48c-5c24-4c19-8f86-1038b8924341' 2024-05-06T03:42:41.894 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 baa3f48c-5c24-4c19-8f86-1038b8924341 2024-05-06T03:42:41.894 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:42:41.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxUThml16vNRAAgXdYhkj842PIcCkTm4h28g== 2024-05-06T03:42:41.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxUThml16vNRAAgXdYhkj842PIcCkTm4h28g=="}' 2024-05-06T03:42:41.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new baa3f48c-5c24-4c19-8f86-1038b8924341 -i td/network-ping/0/new.json 2024-05-06T03:42:42.188 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:42:42.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-05-06T03:42:42.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxUThml16vNRAAgXdYhkj842PIcCkTm4h28g== --osd-uuid baa3f48c-5c24-4c19-8f86-1038b8924341 2024-05-06T03:42:42.228 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.220+0000 7f21c461f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:42.256 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.247+0000 7f21c461f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:42.257 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.248+0000 7f21c461f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:42.257 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.249+0000 7f21c461f640 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-05-06T03:42:42.257 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.249+0000 7f21c461f640 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-05-06T03:42:42.257 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.249+0000 7f21c461f640 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-05-06T03:42:42.296 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:42.287+0000 7f21c461f640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-05-06T03:42:47.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-05-06T03:42:47.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:42:47.806 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:42:47.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:42:47.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:42:48.378 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:42:48.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:42:48.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:42:48.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:42:48.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:42:48.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:42:48.405 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:48.397+0000 7f9db0987640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:48.418 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:48.409+0000 7f9db0987640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:48.419 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:48.410+0000 7f9db0987640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:42:48.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:42:48.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:42:48.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:48.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:48.949 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:48.941+0000 7f9db0987640 -1 Falling back to public interface 2024-05-06T03:42:49.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:50.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:50.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:50.267 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:42:50.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:42:50.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:50.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:50.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:51.310 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:42:51.302+0000 7f9db0987640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:42:51.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:51.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:51.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:42:51.701 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:42:51.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:51.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:52.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:53.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:53.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:53.106 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:42:53.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:42:53.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:53.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:53.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:54.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:54.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:54.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:42:54.527 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:42:54.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:54.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:54.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:55.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:55.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:55.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:42:55.929 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:42:55.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:55.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:56.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:57.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:57.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:57.433 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:42:57.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:42:57.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:57.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:57.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:42:58.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:42:58.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:42:58.840 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:42:58.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:42:58.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:42:58.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:42:59.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:00.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:00.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:00.241 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:43:00.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:43:00.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:00.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:00.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:01.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:01.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:01.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:43:01.645 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:43:01.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:01.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:02.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:03.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:03.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:03.275 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:43:03.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:43:03.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:03.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:03.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:04.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:04.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:04.964 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:43:04.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:43:04.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:04.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:05.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:06.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:06.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:06.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:43:06.486 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:43:06.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:06.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:07.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:08.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:08.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:08.304 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:43:08.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:43:08.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:08.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:09.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:10.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:10.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:10.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:43:10.392 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:43:10.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:10.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:11.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:12.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:12.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:12.585 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:43:12.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:43:12.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:12.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:13.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:14.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:14.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:14.499 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:43:14.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:43:14.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:14.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:43:14.943 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2899577140,v1:127.0.0.1:6803/2899577140] [v2:127.0.0.1:6804/2899577140,v1:127.0.0.1:6805/2899577140] exists,up baa3f48c-5c24-4c19-8f86-1038b8924341 2024-05-06T03:43:14.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:43:14.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:43:14.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:43:14.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:43:14.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e1f0a8c-e295-4b8d-b04f-c4ec0cfdcc9a 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5e1f0a8c-e295-4b8d-b04f-c4ec0cfdcc9a' 2024-05-06T03:43:14.951 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 5e1f0a8c-e295-4b8d-b04f-c4ec0cfdcc9a 2024-05-06T03:43:14.952 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:43:14.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSUThm6DULORAAEkwy/FvTedQI4/9gSROTrw== 2024-05-06T03:43:14.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSUThm6DULORAAEkwy/FvTedQI4/9gSROTrw=="}' 2024-05-06T03:43:14.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e1f0a8c-e295-4b8d-b04f-c4ec0cfdcc9a -i td/network-ping/1/new.json 2024-05-06T03:43:15.439 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:43:15.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-05-06T03:43:15.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSUThm6DULORAAEkwy/FvTedQI4/9gSROTrw== --osd-uuid 5e1f0a8c-e295-4b8d-b04f-c4ec0cfdcc9a 2024-05-06T03:43:15.481 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.472+0000 7f25ef187640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:15.482 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.474+0000 7f25ef187640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:15.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.475+0000 7f25ef187640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:15.484 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.475+0000 7f25ef187640 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-05-06T03:43:15.484 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.475+0000 7f25ef187640 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-05-06T03:43:15.484 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.475+0000 7f25ef187640 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-05-06T03:43:15.516 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:15.508+0000 7f25ef187640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-05-06T03:43:21.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-05-06T03:43:21.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:43:21.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T03:43:21.249 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T03:43:21.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:43:21.798 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T03:43:21.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T03:43:21.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:43:21.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:43:21.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:43:21.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:43:21.826 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:21.817+0000 7f5850215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:21.848 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:21.839+0000 7f5850215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:21.849 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:21.840+0000 7f5850215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:43:22.253 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:43:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:22.381 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:22.373+0000 7f5850215640 -1 Falling back to public interface 2024-05-06T03:43:22.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:23.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:23.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:23.688 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:43:23.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:43:23.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:23.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:24.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:24.742 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:24.733+0000 7f5850215640 -1 osd.1 0 log_to_monitors true 2024-05-06T03:43:25.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:25.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:25.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:43:25.091 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:43:25.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:25.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:25.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:26.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:26.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:26.535 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:43:26.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:43:26.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:26.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:26.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:27.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:27.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:27.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:43:27.958 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:43:27.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:27.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:28.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:29.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:29.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:29.474 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:43:29.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:43:29.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:29.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:29.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:30.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:30.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:30.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:43:30.879 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:43:30.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:30.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:31.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:32.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:32.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:32.354 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:43:32.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:43:32.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:32.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:32.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:33.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:33.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:33.886 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:43:33.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:43:33.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:33.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:34.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:35.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:35.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:35.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:43:35.302 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:43:35.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:35.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:35.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:36.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:36.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:36.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:43:36.710 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:43:36.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:36.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:37.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:38.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:38.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:38.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:43:38.139 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:43:38.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:38.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:39.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:40.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:40.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:40.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:43:40.177 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:43:40.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:40.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:41.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:42.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:42.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:42.050 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:43:42.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:43:42.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:42.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:42.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:43.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:43.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:43.818 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:43:43.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:43:43.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:43.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:44.575 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:45.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:45.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:45.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:43:45.576 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:43:45.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:45.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:46.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:47.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:47.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:47.246 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:43:47.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:43:47.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:47.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:47.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:48.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:48.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:48.691 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:43:48.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:43:48.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:48.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:43:49.105 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1461044138,v1:127.0.0.1:6811/1461044138] [v2:127.0.0.1:6812/1461044138,v1:127.0.0.1:6813/1461044138] exists,up 5e1f0a8c-e295-4b8d-b04f-c4ec0cfdcc9a 2024-05-06T03:43:49.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:43:49.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:43:49.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:43:49.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-05-06T03:43:49.113 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:43:49.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=742a350a-0de1-48c8-b61c-7dffdc9dde0e 2024-05-06T03:43:49.113 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 742a350a-0de1-48c8-b61c-7dffdc9dde0e 2024-05-06T03:43:49.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 742a350a-0de1-48c8-b61c-7dffdc9dde0e' 2024-05-06T03:43:49.113 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:43:49.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1UThmlbIUBxAAE/+gfig+hrRrahmcpzepWw== 2024-05-06T03:43:49.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1UThmlbIUBxAAE/+gfig+hrRrahmcpzepWw=="}' 2024-05-06T03:43:49.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 742a350a-0de1-48c8-b61c-7dffdc9dde0e -i td/network-ping/2/new.json 2024-05-06T03:43:49.585 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:43:49.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-05-06T03:43:49.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1UThmlbIUBxAAE/+gfig+hrRrahmcpzepWw== --osd-uuid 742a350a-0de1-48c8-b61c-7dffdc9dde0e 2024-05-06T03:43:49.626 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.617+0000 7f5996a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:49.633 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.624+0000 7f5996a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.625+0000 7f5996a26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.625+0000 7f5996a26640 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-05-06T03:43:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.625+0000 7f5996a26640 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-05-06T03:43:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.625+0000 7f5996a26640 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-05-06T03:43:49.673 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:49.664+0000 7f5996a26640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-05-06T03:43:55.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-05-06T03:43:55.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:43:55.339 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T03:43:55.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T03:43:55.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:43:55.890 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T03:43:55.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T03:43:55.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:43:55.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:43:55.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:43:55.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:43:55.918 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:55.909+0000 7fcf1ee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:55.923 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:55.914+0000 7fcf1ee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:55.924 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:55.916+0000 7fcf1ee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:43:56.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T03:43:56.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:43:56.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T03:43:56.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:43:56.371 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:43:56.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:43:56.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:56.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:43:56.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:56.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:43:56.462 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:56.454+0000 7fcf1ee0f640 -1 Falling back to public interface 2024-05-06T03:43:56.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:57.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:57.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:57.806 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:43:57.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:43:57.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:57.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:43:58.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:43:58.860 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:43:58.851+0000 7fcf1ee0f640 -1 osd.2 0 log_to_monitors true 2024-05-06T03:43:59.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:43:59.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:43:59.211 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:43:59.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:43:59.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:43:59.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:43:59.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:00.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:00.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:00.648 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:44:00.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:44:00.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:00.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:01.046 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:02.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:02.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:02.048 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:44:02.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:44:02.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:02.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:02.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:03.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:03.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:03.457 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:44:03.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:44:03.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:03.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:03.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:04.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:04.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:04.959 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:44:04.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:44:04.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:04.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:05.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:06.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:06.424 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:44:06.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:06.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:44:06.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:06.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:06.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:07.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:07.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:07.843 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:44:07.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:44:07.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:07.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:08.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:09.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:09.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:09.254 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:44:09.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:44:09.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:09.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:09.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:10.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:10.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:10.765 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:44:10.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:44:10.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:10.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:11.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:12.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:12.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:12.273 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:44:12.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:44:12.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:12.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:13.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:14.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:14.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:14.012 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:44:14.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:44:14.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:14.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:14.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:15.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:15.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:15.616 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:44:15.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:44:15.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:15.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:16.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:17.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:17.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:17.380 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:44:17.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:44:17.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:17.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:18.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:19.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:19.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:19.196 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:44:19.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:44:19.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:19.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:20.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:21.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:21.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:21.094 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:44:21.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:44:21.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:21.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:21.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:44:22.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:44:22.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:44:22.668 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:44:22.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:44:22.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:44:22.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:44:23.091 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4021442071,v1:127.0.0.1:6819/4021442071] [v2:127.0.0.1:6820/4021442071,v1:127.0.0.1:6821/4021442071] exists,up 742a350a-0de1-48c8-b61c-7dffdc9dde0e 2024-05-06T03:44:23.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:44:23.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:44:23.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:44:23.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-05-06T03:44:28.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-05-06T03:44:28.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-05-06T03:44:28.592 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'foo' already exists 2024-05-06T03:44:28.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T03:44:29.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-05-06T03:44:29.653 INFO:tasks.workunit.client.0.smithi163.stdout:hints = 1 2024-05-06T03:44:29.654 INFO:tasks.workunit.client.0.smithi163.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-05-06T03:44:29.654 INFO:tasks.workunit.client.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_45501 2024-05-06T03:44:29.654 INFO:tasks.workunit.client.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-06T03:44:29.654 INFO:tasks.workunit.client.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-05-06T03:44:30.654 INFO:tasks.workunit.client.0.smithi163.stdout: 1 16 63 47 0.183571 0.183594 0.260469 0.276582 2024-05-06T03:44:31.654 INFO:tasks.workunit.client.0.smithi163.stdout: 2 16 119 103 0.201145 0.21875 0.270426 0.284859 2024-05-06T03:44:32.654 INFO:tasks.workunit.client.0.smithi163.stdout: 3 16 183 167 0.217418 0.25 0.229108 0.274191 2024-05-06T03:44:33.654 INFO:tasks.workunit.client.0.smithi163.stdout: 4 16 247 231 0.225558 0.25 0.233861 0.268826 2024-05-06T03:44:34.654 INFO:tasks.workunit.client.0.smithi163.stdout: 5 16 311 295 0.23044 0.25 0.235233 0.26365 2024-05-06T03:44:35.654 INFO:tasks.workunit.client.0.smithi163.stdout: 6 16 375 359 0.233694 0.25 0.234873 0.261617 2024-05-06T03:44:36.657 INFO:tasks.workunit.client.0.smithi163.stdout: 7 16 439 423 0.236018 0.25 0.232963 0.259189 2024-05-06T03:44:37.654 INFO:tasks.workunit.client.0.smithi163.stdout: 8 16 503 487 0.237762 0.25 0.251194 0.258073 2024-05-06T03:44:38.655 INFO:tasks.workunit.client.0.smithi163.stdout: 9 16 559 543 0.235646 0.21875 0.251275 0.260837 2024-05-06T03:44:39.655 INFO:tasks.workunit.client.0.smithi163.stdout: 10 16 639 623 0.243329 0.3125 0.17666 0.255673 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Total time run: 10.1738 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Total writes made: 639 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Write size: 4096 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Object size: 4096 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Bandwidth (MB/sec): 0.245345 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Stddev Bandwidth: 0.0330457 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Max bandwidth (MB/sec): 0.3125 2024-05-06T03:44:40.655 INFO:tasks.workunit.client.0.smithi163.stdout:Min bandwidth (MB/sec): 0.183594 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Average IOPS: 62 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Stddev IOPS: 8.45971 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Max IOPS: 80 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Min IOPS: 47 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Average Latency(s): 0.25361 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Stddev Latency(s): 0.0409919 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Max latency(s): 0.561989 2024-05-06T03:44:40.656 INFO:tasks.workunit.client.0.smithi163.stdout:Min latency(s): 0.134258 2024-05-06T03:44:40.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-05-06T03:44:50.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-05-06T03:44:50.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:44:50.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:44:51.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:44:51.220 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:44:51.220 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T03:44:51.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:44:51.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:44:51.220 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:44:51.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-05-06T03:44:51.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-05-06T03:44:51.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-05-06T03:44:51.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:44:51.335 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:44:51.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-05-06T03:44:51.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-05-06T03:44:51.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-05-06T03:44:51.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:44:51.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:44:51.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-05-06T03:44:51.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-05-06T03:44:51.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509449' 2024-05-06T03:44:51.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:44:51.563 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-05-06T03:44:51.563 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:44:51.563 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:44:51.564 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-05-06T03:44:51.564 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:44:51.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-05-06T03:44:51.565 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836505 2024-05-06T03:44:51.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-05-06T03:44:51.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:44:51.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836505 2024-05-06T03:44:51.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:44:52.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:44:52.998 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:44:53.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836505 2024-05-06T03:44:53.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:44:53.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-05-06T03:44:53.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:44:53.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:44:53.433 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-05-06T03:44:53.433 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:44:53.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-05-06T03:44:53.434 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672978 2024-05-06T03:44:53.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-05-06T03:44:53.434 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:44:53.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-05-06T03:44:53.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:44:53.833 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-05-06T03:44:53.833 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:44:53.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:44:53.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-05-06T03:44:53.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:44:53.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-05-06T03:44:53.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-05-06T03:44:53.836 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509449 2024-05-06T03:44:53.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:44:54.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509449 2024-05-06T03:44:54.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:44:54.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:44:54.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 2024-05-06T03:44:54.310 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:44:54.310 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 1000, 2024-05-06T03:44:54.310 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:44:54.310 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:44:54.318 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.318 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:44:54.320 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:44:54.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-05-06T03:44:54.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-05-06T03:44:54.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:44:54.324 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:44:54.324 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:44:54.324 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:44:54.325 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:44:54.325 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:44:54.325 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:44:54.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:44:54.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:44:54.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 2024-05-06T03:44:54.396 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:44:54.396 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 1000, 2024-05-06T03:44:54.396 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:44:54.396 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:44:54.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:44:54.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:44:54.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-05-06T03:44:54.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-05-06T03:44:54.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:44:54.410 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:44:54.410 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:44:54.410 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:44:54.411 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:44:54.411 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:44:54.411 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:44:54.411 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:44:54.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:44:54.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 0 2024-05-06T03:44:54.488 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:44:54.488 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 0, 2024-05-06T03:44:54.488 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [ 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:54 2024", 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.284, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.338, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.310 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.309, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.471, 2024-05-06T03:44:54.489 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.471 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.309, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.471, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.471 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.266 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:54 2024", 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.490 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.266, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.319, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.290 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.283, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.447, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.447 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.283, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.447, 2024-05-06T03:44:54.491 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.447 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.253 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:53 2024", 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.276, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.315, 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.302 2024-05-06T03:44:54.492 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.316, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.464, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.464 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.316, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.464, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.464 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.230 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:53 2024", 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.493 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.245, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.290, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.289 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.293, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.493, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.493 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.494 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.293, 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.493, 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.493 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.212 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T03:44:54.495 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:44:54.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:44:54.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-05-06T03:44:54.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-05-06T03:44:54.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:44:54.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:44:54.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 0 2024-05-06T03:44:54.574 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:44:54.574 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 0, 2024-05-06T03:44:54.574 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [ 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:48 2024", 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.347, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.349, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.337 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.275, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.224, 2024-05-06T03:44:54.575 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.224 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.416, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.481, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.481 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.304 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:52 2024", 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.576 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.277, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.312, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.343 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.231, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.222, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.222 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.338, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.441, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.441 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.577 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.256 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:48 2024", 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.333, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.340, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.342 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.578 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.248, 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.215, 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.215 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.393, 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.451, 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.451 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.579 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.338 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:49 2024", 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.323, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.341, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.311 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.580 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.275, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.249, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.249 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.366, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.471, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.471 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.271 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:52 2024", 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.581 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.276, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.320, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.338 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.246, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.246, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.246 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.296, 2024-05-06T03:44:54.582 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.442, 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.442 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.296 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:48 2024", 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.583 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.584 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.332, 2024-05-06T03:44:54.584 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.332, 2024-05-06T03:44:54.584 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.314 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.272, 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.205, 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.189 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.391, 2024-05-06T03:44:54.585 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.457, 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.499 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.313 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:52 2024", 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:44:54.586 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.260, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.295, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.322 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.213, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.213, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.213 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.310, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.465, 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.465 2024-05-06T03:44:54.587 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.272 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:49 2024", 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.298, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.321, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.291 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.588 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.256, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.234, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.234 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.344, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.447, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.447 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.254 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.589 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:48 2024", 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.305, 2024-05-06T03:44:54.590 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.316, 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.303 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.227, 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.182, 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.172 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.413, 2024-05-06T03:44:54.591 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.470, 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.477 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.273 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:53 2024", 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.592 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.276, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.315, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.302 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.236, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.229, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.176 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.316, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.464, 2024-05-06T03:44:54.593 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.464 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.236 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:52 2024", 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.233, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.285, 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.302 2024-05-06T03:44:54.594 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.205, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.205, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.205 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.263, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.481, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.481 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.225 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:44:53 2024", 2024-05-06T03:44:54.595 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.245, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.290, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.289 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.219, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.195, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.164 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:44:54.596 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.293, 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.493, 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.493 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.219 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T03:44:54.597 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:44:54.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-05-06T03:45:14.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-05-06T03:45:14.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:45:14.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:45:15.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:45:15.026 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:45:15.026 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T03:45:15.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:45:15.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:45:15.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:45:15.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-05-06T03:45:15.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-05-06T03:45:15.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-05-06T03:45:15.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:45:15.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:45:15.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-05-06T03:45:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-05-06T03:45:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672985' 2024-05-06T03:45:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:45:15.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:45:15.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-05-06T03:45:15.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-05-06T03:45:15.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509456' 2024-05-06T03:45:15.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:45:15.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-05-06T03:45:15.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:45:15.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:45:15.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-05-06T03:45:15.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:45:15.370 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836512 2024-05-06T03:45:15.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-05-06T03:45:15.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-05-06T03:45:15.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:45:15.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836512 2024-05-06T03:45:15.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:45:16.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:45:16.803 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:45:17.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836512 2024-05-06T03:45:17.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:45:17.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-05-06T03:45:17.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:45:17.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:45:17.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-05-06T03:45:17.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:45:17.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-05-06T03:45:17.298 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672985 2024-05-06T03:45:17.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-05-06T03:45:17.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:45:17.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-05-06T03:45:17.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:45:17.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-05-06T03:45:17.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:45:17.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:45:17.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-05-06T03:45:17.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:45:17.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-05-06T03:45:17.725 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509456 2024-05-06T03:45:17.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-05-06T03:45:17.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:45:18.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509456 2024-05-06T03:45:18.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:45:18.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:45:18.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 2024-05-06T03:45:18.198 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:45:18.199 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 1000, 2024-05-06T03:45:18.199 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:45:18.199 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:45:18.206 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.206 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:45:18.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:45:18.209 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.209 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-05-06T03:45:18.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:45:18.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 2024-05-06T03:45:18.284 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:45:18.284 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 1000, 2024-05-06T03:45:18.284 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:45:18.284 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:45:18.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.292 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:45:18.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:45:18.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.295 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-05-06T03:45:18.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-05-06T03:45:18.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:45:18.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:45:18.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:45:18.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:45:18.298 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:45:18.299 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:45:18.299 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:45:18.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:45:18.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:45:18.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 0 2024-05-06T03:45:18.377 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:45:18.377 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 0, 2024-05-06T03:45:18.377 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [ 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:14 2024", 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.330, 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.329, 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.322 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.378 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.385, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.489, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.489 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.385, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.489, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.489 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.238 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:14 2024", 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.379 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.329, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.310, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.302 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.453, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.453, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.453 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.453, 2024-05-06T03:45:18.380 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.453, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.453 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.261 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:18 2024", 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.262, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.311, 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.298 2024-05-06T03:45:18.381 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.351, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.439, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.464 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.351, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.439, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.464 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.290 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.382 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:18 2024", 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.251, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.300, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.286 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.310, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.482, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.493 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.383 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.310, 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.482, 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.493 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.310 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T03:45:18.384 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:45:18.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:45:18.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-05-06T03:45:18.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.389 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-05-06T03:45:18.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:45:18.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:45:18.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:45:18.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:45:18.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:45:18.393 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:45:18.393 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:45:18.393 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:45:18.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:45:18.394 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:45:18.394 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 0 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 0, 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [ 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:13 2024", 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:45:18.463 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.363, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.328, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.336 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.270, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.219, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.219 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.418, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.442, 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.481 2024-05-06T03:45:18.464 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.238 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:13 2024", 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.360, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.309, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.308 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.309, 2024-05-06T03:45:18.465 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.198, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.189 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.433, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.433, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.457 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.253 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:12 2024", 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:45:18.466 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.349, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.295, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.314 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.277, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.213, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.213 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.470, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.470, 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.470 2024-05-06T03:45:18.467 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.247 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:13 2024", 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.345, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.323, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.335 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.291, 2024-05-06T03:45:18.468 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.234, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.215 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.408, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.460, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.460 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.284 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:13 2024", 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:45:18.469 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.342, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.299, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.288 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.271, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.212, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.164 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.405, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.482, 2024-05-06T03:45:18.470 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.493 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.254 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:12 2024", 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.340, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.295, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.325 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.471 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.266, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.227, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.227 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.490, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.490, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.490 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.269 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:14 2024", 2024-05-06T03:45:18.472 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.330, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.329, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.322 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.291, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.266, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.249 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.473 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.385, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.489, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.489 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.299 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:14 2024", 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.474 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.329, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.310, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.302 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.270, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.227, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.227 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.453, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.453, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.453 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.475 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.288 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:12 2024", 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.320, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.287, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.325 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.476 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.252, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.223, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.222 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.421, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.421, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.441 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.232 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:13 2024", 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.477 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.324, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.314, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.300 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.249, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.230, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.176 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.478 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.431, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.439, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.464 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.351 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:13 2024", 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.314, 2024-05-06T03:45:18.479 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.293, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.299 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.224, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.180, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.172 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.367, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.482, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.482 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.270 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:45:18.480 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:45:12 2024", 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.291, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.272, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.294 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.224, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.205, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.205 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.439, 2024-05-06T03:45:18.481 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.439, 2024-05-06T03:45:18.482 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.481 2024-05-06T03:45:18.482 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:45:18.482 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.217 2024-05-06T03:45:18.482 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T03:45:18.482 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T03:45:18.482 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:45:18.483 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.483 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:45:18.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-05-06T03:45:18.483 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:45:18.483 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-05-06T03:45:18.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:45:18.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-05-06T03:46:08.478 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-05-06T03:46:08.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:46:08.479 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:46:08.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:46:08.914 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:46:08.915 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T03:46:08.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:46:08.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:46:08.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:46:09.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836525 2024-05-06T03:46:09.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836525 2024-05-06T03:46:09.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525' 2024-05-06T03:46:09.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:46:09.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:46:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-05-06T03:46:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-05-06T03:46:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525 1-42949672998' 2024-05-06T03:46:09.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:46:09.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:46:09.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509469 2024-05-06T03:46:09.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509469 2024-05-06T03:46:09.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525 1-42949672998 2-64424509469' 2024-05-06T03:46:09.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:46:09.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836525 2024-05-06T03:46:09.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:46:09.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:46:09.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836525 2024-05-06T03:46:09.261 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:46:09.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836525 2024-05-06T03:46:09.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-05-06T03:46:09.262 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836525 2024-05-06T03:46:09.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:46:09.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836522 -lt 21474836525 2024-05-06T03:46:09.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:46:10.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:46:10.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:46:11.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836525 2024-05-06T03:46:11.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:46:11.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-05-06T03:46:11.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:46:11.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:46:11.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-05-06T03:46:11.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:46:11.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-05-06T03:46:11.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-05-06T03:46:11.130 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672998 2024-05-06T03:46:11.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:46:11.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-05-06T03:46:11.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:46:11.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509469 2024-05-06T03:46:11.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:46:11.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:46:11.637 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509469 2024-05-06T03:46:11.637 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:46:11.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509469 2024-05-06T03:46:11.638 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509469 2024-05-06T03:46:11.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-05-06T03:46:11.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:46:12.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509469 2024-05-06T03:46:12.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:46:12.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 2024-05-06T03:46:12.175 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:46:12.175 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 1000, 2024-05-06T03:46:12.176 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:46:12.176 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:46:12.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:46:12.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:46:12.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.187 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-05-06T03:46:12.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-05-06T03:46:12.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:46:12.190 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:46:12.190 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:46:12.190 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:46:12.190 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:12.190 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.191 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.191 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:46:12.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:46:12.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 2024-05-06T03:46:12.263 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:46:12.263 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 1000, 2024-05-06T03:46:12.263 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:46:12.263 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:46:12.271 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.271 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:46:12.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:46:12.274 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.274 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-05-06T03:46:12.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-05-06T03:46:12.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:46:12.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 0 2024-05-06T03:46:12.360 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 0, 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [ 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:09 2024", 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.345, 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.328, 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.335 2024-05-06T03:46:12.361 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.427, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.523, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.523 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.427, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.523, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.523 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.295 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.362 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.336, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.314, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.313 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.378, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.468, 2024-05-06T03:46:12.363 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.482 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.378, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.468, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.482 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.278 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:09 2024", 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.364 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.328, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.318, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.321 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.415, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.435, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.453 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.415, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.435, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.453 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.245 2024-05-06T03:46:12.365 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.318, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.317, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.321 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.378, 2024-05-06T03:46:12.366 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.480, 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.489 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.378, 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.480, 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.489 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.260 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T03:46:12.367 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:46:12.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:46:12.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-05-06T03:46:12.372 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.372 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-05-06T03:46:12.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:46:12.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:46:12.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 0 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 0, 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [ 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:07 2024", 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:46:12.450 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.318, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.340, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.324 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.265, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.265, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.219 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.418, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.472, 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.502 2024-05-06T03:46:12.451 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.380 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.336, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.314, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.313 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.452 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.306, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.227, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.212 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.378, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.468, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.482 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.322 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:04 2024", 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.453 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.299, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.318, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.330 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.257, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.241, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.202 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.321, 2024-05-06T03:46:12.454 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.523, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.523 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.383 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.329, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.317, 2024-05-06T03:46:12.455 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.328 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.272, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.248, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.235 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.433, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.462, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.530 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.272 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.456 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:07 2024", 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.308, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.323, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.310 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.278, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.243, 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.223 2024-05-06T03:46:12.457 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.368, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.433, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.484 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.367 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 1, 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.458 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.318, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.317, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.321 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.250, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.229, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.215 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.378, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.480, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.489 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.459 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.344 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:07 2024", 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.297, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.320, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.310 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.460 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.250, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.229, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.226 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.386, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.447, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.470 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.350 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.461 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.320, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.299, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.316 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.259, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.237, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.220 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.462 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.392, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.439, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.481 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.259 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:04 2024", 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 0, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 2, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.296, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.309, 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.317 2024-05-06T03:46:12.463 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.277, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.239, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.183 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.335, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.435, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.453 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.367 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.464 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "front", 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.311, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.301, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.311 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.239, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.228, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.216 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.407, 2024-05-06T03:46:12.465 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.481, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.481 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.239 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:07 2024", 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 2, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.261, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.295, 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.293 2024-05-06T03:46:12.466 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.213, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.213, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.187 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.342, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.412, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.483 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.338 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T03:46:12.467 INFO:tasks.workunit.client.0.smithi163.stdout: "last update": "Mon May 6 03:46:08 2024", 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "stale": false, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "from osd": 1, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "to osd": 0, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "interface": "back", 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "average": { 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.280, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.273, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.290 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "min": { 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.220, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.220, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.202 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.468 INFO:tasks.workunit.client.0.smithi163.stdout: "max": { 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: "1min": 0.373, 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: "5min": 0.416, 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: "15min": 0.485 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: "last": 0.220 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T03:46:12.469 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:46:12.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-05-06T03:46:12.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-05-06T03:46:12.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-05-06T03:46:12.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mgr.x.asok 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:46:12.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-mgr.x.asok dump_osd_network 99 2024-05-06T03:46:12.539 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:46:12.539 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 99, 2024-05-06T03:46:12.539 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:46:12.539 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:46:12.547 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.547 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-05-06T03:46:12.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-05-06T03:46:12.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-osd.0.asok 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-05-06T03:46:12.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40033/ceph-osd.0.asok dump_osd_network 98 2024-05-06T03:46:12.631 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T03:46:12.632 INFO:tasks.workunit.client.0.smithi163.stdout: "threshold": 98, 2024-05-06T03:46:12.632 INFO:tasks.workunit.client.0.smithi163.stdout: "entries": [] 2024-05-06T03:46:12.632 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T03:46:12.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-05-06T03:46:12.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-05-06T03:46:12.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-05-06T03:46:12.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:46:12.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:46:12.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:46:12.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:46:12.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:46:12.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:46:12.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:46:12.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:46:12.754 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:46:12.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:46:12.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:46:12.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:46:12.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:46:12.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:46:12.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:46:12.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-05-06T03:46:12.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:46:12.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40033 2024-05-06T03:46:12.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:46:12.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:46:12.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:46:12.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:46:12.779 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:46:12.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:46:12.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:46:12.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:46:12.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:46:12.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:46:12.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:46:12.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:46:12.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:46:12.783 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:46:12.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:46:12.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:46:12.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-05-06T03:46:12.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:46:12.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40033 2024-05-06T03:46:12.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:46:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:46:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-05-06T03:46:12.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:46:12.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:12.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.40033 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-05-06T03:46:12.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-05-06T03:46:12.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-05-06T03:46:13.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:46:13.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:46:13.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:46:13.250 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:46:13.250 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:13.250 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:13.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:46:13.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:46:13.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:46:13.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:46:13.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:46:13.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:46:13.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:46:13.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mon.a.asok 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:46:13.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40033/ceph-mon.a.asok config get fsid 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:13.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40033/ceph-mon.a.asok 2024-05-06T03:46:13.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:46:13.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40033/ceph-mon.a.asok config get mon_host 2024-05-06T03:46:13.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-05-06T03:46:13.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-05-06T03:46:13.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:46:13.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:46:13.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:46:13.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-05-06T03:46:13.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:46:14.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:46:14.441 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:46:14.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-05-06T03:46:14.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-05-06T03:46:14.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:46:14.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:46:14.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27f966b5-99af-416e-b9b8-7b3c2927c0c6 2024-05-06T03:46:14.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 27f966b5-99af-416e-b9b8-7b3c2927c0c6' 2024-05-06T03:46:14.474 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 27f966b5-99af-416e-b9b8-7b3c2927c0c6 2024-05-06T03:46:14.474 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:46:14.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGUjhmN26gHBAAJ+d22dr5PuMmZvF0SUSWMQ== 2024-05-06T03:46:14.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGUjhmN26gHBAAJ+d22dr5PuMmZvF0SUSWMQ=="}' 2024-05-06T03:46:14.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27f966b5-99af-416e-b9b8-7b3c2927c0c6 -i td/network-ping/0/new.json 2024-05-06T03:46:14.776 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:46:14.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-05-06T03:46:14.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGUjhmN26gHBAAJ+d22dr5PuMmZvF0SUSWMQ== --osd-uuid 27f966b5-99af-416e-b9b8-7b3c2927c0c6 2024-05-06T03:46:14.816 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.806+0000 7f27b081f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:14.836 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.826+0000 7f27b081f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:14.837 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.827+0000 7f27b081f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:14.837 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.828+0000 7f27b081f640 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-05-06T03:46:14.837 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.828+0000 7f27b081f640 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-05-06T03:46:14.837 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.828+0000 7f27b081f640 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-05-06T03:46:14.876 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:14.866+0000 7f27b081f640 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-05-06T03:46:20.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-05-06T03:46:20.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:46:20.330 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:46:20.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:46:20.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:46:20.879 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:46:20.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:46:20.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:46:20.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:46:20.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:46:20.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:46:20.906 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:20.897+0000 7ff9aa968640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:20.908 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:20.899+0000 7ff9aa968640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:20.909 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:20.900+0000 7ff9aa968640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:21.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:46:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:46:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:46:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:46:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:46:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:46:21.307 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:46:21.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:21.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:21.427 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:21.418+0000 7ff9aa968640 -1 Falling back to public interface 2024-05-06T03:46:21.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:22.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:22.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:22.741 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:46:22.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:46:22.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:22.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:23.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:23.824 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:23.814+0000 7ff9aa968640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:46:24.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:24.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:24.175 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:46:24.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:46:24.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:24.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:24.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:25.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:25.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:25.726 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:46:25.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:46:25.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:25.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:26.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:27.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:27.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:27.151 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:46:27.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:46:27.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:27.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:27.575 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:28.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:28.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:28.577 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:46:28.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:46:28.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:28.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:29.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:30.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:30.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:30.010 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:46:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:46:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:30.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:31.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:31.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:31.588 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:46:31.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:46:31.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:31.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:32.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:33.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:33.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:33.009 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:46:33.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:46:33.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:33.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:33.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:34.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:34.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:34.537 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:46:34.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:46:34.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:34.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:34.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:35.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:35.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:35.963 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:46:35.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:46:35.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:35.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:36.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:37.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:37.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:37.987 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:46:37.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:46:37.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:37.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:38.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:39.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:39.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:39.704 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:46:39.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:46:39.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:39.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:40.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:41.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:41.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:41.440 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:46:41.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:46:41.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:41.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:42.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:43.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:43.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:43.297 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:46:43.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:46:43.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:43.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:44.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:45.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:45.039 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:46:45.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:46:45.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:45.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:46.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:47.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:47.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:47.033 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:46:47.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:46:47.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:47.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:46:47.438 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1572079805,v1:127.0.0.1:6803/1572079805] [v2:127.0.0.1:6804/1572079805,v1:127.0.0.1:6805/1572079805] exists,up 27f966b5-99af-416e-b9b8-7b3c2927c0c6 2024-05-06T03:46:47.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-05-06T03:46:47.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:46:47.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71775f81-0a08-484b-b15a-0e88352abb07 2024-05-06T03:46:47.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 71775f81-0a08-484b-b15a-0e88352abb07' 2024-05-06T03:46:47.446 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 71775f81-0a08-484b-b15a-0e88352abb07 2024-05-06T03:46:47.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:46:47.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnUjhmrF3bGhAASycDMWplvSRAbqwn0yzFfQ== 2024-05-06T03:46:47.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnUjhmrF3bGhAASycDMWplvSRAbqwn0yzFfQ=="}' 2024-05-06T03:46:47.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71775f81-0a08-484b-b15a-0e88352abb07 -i td/network-ping/1/new.json 2024-05-06T03:46:47.958 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:46:47.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-05-06T03:46:47.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnUjhmrF3bGhAASycDMWplvSRAbqwn0yzFfQ== --osd-uuid 71775f81-0a08-484b-b15a-0e88352abb07 2024-05-06T03:46:47.999 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:47.990+0000 7f64eba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:48.004 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:47.995+0000 7f64eba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:48.005 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:47.996+0000 7f64eba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:48.006 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:47.997+0000 7f64eba0f640 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-05-06T03:46:48.006 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:47.997+0000 7f64eba0f640 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-05-06T03:46:48.006 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:47.997+0000 7f64eba0f640 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-05-06T03:46:48.045 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:48.035+0000 7f64eba0f640 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-05-06T03:46:53.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-05-06T03:46:53.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:46:53.831 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T03:46:53.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T03:46:53.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:46:54.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T03:46:54.389 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T03:46:54.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:46:54.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:46:54.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:46:54.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:46:54.417 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:54.408+0000 7f1b6e611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:54.419 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:54.410+0000 7f1b6e611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:54.420 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:54.411+0000 7f1b6e611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:46:54.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T03:46:54.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:46:54.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T03:46:54.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:46:54.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:46:54.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:54.808 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:46:54.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:46:54.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:54.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:46:54.947 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:54.938+0000 7f1b6e611640 -1 Falling back to public interface 2024-05-06T03:46:55.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:56.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:56.243 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:46:56.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:56.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:46:56.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:56.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:46:56.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:57.280 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:46:57.271+0000 7f1b6e611640 -1 osd.1 0 log_to_monitors true 2024-05-06T03:46:57.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:57.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:57.711 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:46:57.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:46:57.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:57.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:46:58.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:46:59.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:46:59.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:46:59.109 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:46:59.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:46:59.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:46:59.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:46:59.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:00.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:00.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:00.518 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:47:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:47:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:00.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:01.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:01.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:01.947 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:47:01.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:47:01.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:01.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:02.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:03.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:03.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:03.461 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:47:03.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:47:03.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:03.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:03.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:04.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:04.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:04.968 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:47:04.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:47:04.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:04.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:05.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:06.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:06.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:06.422 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:47:06.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:47:06.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:06.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:06.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:07.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:07.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:07.828 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:47:07.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:47:07.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:07.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:08.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:09.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:09.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:09.243 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:47:09.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:47:09.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:09.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:09.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:10.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:10.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:10.645 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:47:10.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:47:10.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:10.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:11.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:12.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:12.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:12.611 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:47:12.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:47:12.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:12.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:13.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:14.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:14.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:14.054 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:47:14.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:47:14.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:14.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:14.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:15.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:15.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:15.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:47:15.840 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:47:15.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:15.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:16.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:17.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:17.548 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:47:17.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:17.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:47:17.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:17.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:19.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:19.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:19.367 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:47:19.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:47:19.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:19.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:20.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:21.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:21.220 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:47:21.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:21.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:47:21.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:21.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:47:21.616 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3476872480,v1:127.0.0.1:6811/3476872480] [v2:127.0.0.1:6812/3476872480,v1:127.0.0.1:6813/3476872480] exists,up 71775f81-0a08-484b-b15a-0e88352abb07 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-05-06T03:47:21.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:47:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:47:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-05-06T03:47:21.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:47:21.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd844fcd-e4c3-487b-b8dd-9f70a4a3af33 2024-05-06T03:47:21.624 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 fd844fcd-e4c3-487b-b8dd-9f70a4a3af33 2024-05-06T03:47:21.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fd844fcd-e4c3-487b-b8dd-9f70a4a3af33' 2024-05-06T03:47:21.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:47:21.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJUjhmz9eJJRAAEFTiT1y2X1VKSoAmE7RVcg== 2024-05-06T03:47:21.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJUjhmz9eJJRAAEFTiT1y2X1VKSoAmE7RVcg=="}' 2024-05-06T03:47:21.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd844fcd-e4c3-487b-b8dd-9f70a4a3af33 -i td/network-ping/2/new.json 2024-05-06T03:47:22.183 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:47:22.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-05-06T03:47:22.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJUjhmz9eJJRAAEFTiT1y2X1VKSoAmE7RVcg== --osd-uuid fd844fcd-e4c3-487b-b8dd-9f70a4a3af33 2024-05-06T03:47:22.224 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.216+0000 7f8c44df3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:47:22.226 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.217+0000 7f8c44df3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:47:22.227 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.218+0000 7f8c44df3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:47:22.228 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.219+0000 7f8c44df3640 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-05-06T03:47:22.228 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.219+0000 7f8c44df3640 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-05-06T03:47:22.228 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.219+0000 7f8c44df3640 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-05-06T03:47:22.273 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:22.264+0000 7f8c44df3640 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-05-06T03:47:28.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-05-06T03:47:28.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:47:28.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T03:47:28.262 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T03:47:28.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:47:28.813 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T03:47:28.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T03:47:28.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=84068c81-c248-43e4-a659-649116836c25 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40033/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:47:28.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:47:28.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:47:28.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:47:28.842 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:28.833+0000 7fc5097dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:47:28.844 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:28.836+0000 7fc5097dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:47:28.846 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:28.837+0000 7fc5097dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:47:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T03:47:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:47:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T03:47:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:47:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:47:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:29.259 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:47:29.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:47:29.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:29.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:29.372 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:29.364+0000 7fc5097dc640 -1 Falling back to public interface 2024-05-06T03:47:29.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:30.678 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:30.679 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:47:30.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:30.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:47:30.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:30.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:31.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:31.838 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:47:31.830+0000 7fc5097dc640 -1 osd.2 0 log_to_monitors true 2024-05-06T03:47:32.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:32.103 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:47:32.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:32.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:47:32.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:32.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:32.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:33.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:33.614 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:47:33.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:33.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:47:33.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:33.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:34.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:35.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:35.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:35.206 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:47:35.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:47:35.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:35.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:35.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:36.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:36.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:36.745 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:47:36.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:47:36.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:36.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:37.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:38.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:38.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:38.153 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:47:38.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:47:38.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:38.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:38.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:39.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:39.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:39.651 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:47:39.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:47:39.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:39.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:40.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:41.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:41.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:41.066 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:47:41.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:47:41.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:41.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:41.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:42.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:42.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:42.663 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:47:42.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:47:42.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:42.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:43.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:44.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:44.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:44.114 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:47:44.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:47:44.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:44.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:44.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:45.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:45.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:45.531 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:47:45.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:47:45.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:45.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:46.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:47.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:47.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:47.387 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:47:47.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:47:47.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:47.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:48.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:49.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:49.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:49.386 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:47:49.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:47:49.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:49.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:49.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:50.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:50.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:50.873 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:47:50.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:47:50.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:50.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:51.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:52.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:52.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:52.913 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:47:52.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:47:52.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:52.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:53.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:47:54.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:47:54.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:47:54.846 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:47:54.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:47:54.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:47:54.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:47:55.338 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3478659176,v1:127.0.0.1:6819/3478659176] [v2:127.0.0.1:6820/3478659176,v1:127.0.0.1:6821/3478659176] exists,up fd844fcd-e4c3-487b-b8dd-9f70a4a3af33 2024-05-06T03:47:55.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:47:55.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:47:55.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:47:55.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-05-06T03:48:00.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-05-06T03:48:00.873 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'dc1' already exists 2024-05-06T03:48:00.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-05-06T03:48:01.381 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'dc2' already exists 2024-05-06T03:48:01.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-05-06T03:48:01.882 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'dc3' already exists 2024-05-06T03:48:01.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-05-06T03:48:02.571 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'rack1' already exists 2024-05-06T03:48:02.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-05-06T03:48:03.125 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'rack2' already exists 2024-05-06T03:48:03.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-05-06T03:48:03.621 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'rack3' already exists 2024-05-06T03:48:03.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-05-06T03:48:04.124 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'host1' already exists 2024-05-06T03:48:04.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-05-06T03:48:04.705 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'host2' already exists 2024-05-06T03:48:04.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-05-06T03:48:05.349 INFO:tasks.workunit.client.0.smithi163.stderr:bucket 'host3' already exists 2024-05-06T03:48:05.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-05-06T03:48:05.952 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-05-06T03:48:05.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-05-06T03:48:06.496 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-05-06T03:48:06.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-05-06T03:48:07.165 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-05-06T03:48:07.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-05-06T03:48:07.734 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-05-06T03:48:07.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-05-06T03:48:08.317 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-05-06T03:48:08.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-05-06T03:48:08.861 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-05-06T03:48:08.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-05-06T03:48:09.380 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-05-06T03:48:09.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-05-06T03:48:09.932 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-05-06T03:48:09.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-05-06T03:48:10.501 INFO:tasks.workunit.client.0.smithi163.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-05-06T03:48:10.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-05-06T03:48:10.977 INFO:tasks.workunit.client.0.smithi163.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-05-06T03:48:10.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-05-06T03:48:11.514 INFO:tasks.workunit.client.0.smithi163.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-05-06T03:48:11.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-05-06T03:48:12.076 INFO:tasks.workunit.client.0.smithi163.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-05-06T03:48:12.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-05-06T03:48:12.584 INFO:tasks.workunit.client.0.smithi163.stderr:rule myrule already exists 2024-05-06T03:48:12.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-05-06T03:48:12.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-05-06T03:48:13.110 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'foo' already exists 2024-05-06T03:48:13.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T03:48:14.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-05-06T03:48:14.202 INFO:tasks.workunit.client.0.smithi163.stdout:hints = 1 2024-05-06T03:48:14.202 INFO:tasks.workunit.client.0.smithi163.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-05-06T03:48:14.202 INFO:tasks.workunit.client.0.smithi163.stdout:Object prefix: benchmark_data_smithi163_52263 2024-05-06T03:48:14.203 INFO:tasks.workunit.client.0.smithi163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-05-06T03:48:14.203 INFO:tasks.workunit.client.0.smithi163.stdout: 0 0 0 0 0 0 - 0 2024-05-06T03:48:15.202 INFO:tasks.workunit.client.0.smithi163.stdout: 1 16 49 33 0.128894 0.128906 0.260228 0.437743 2024-05-06T03:48:16.202 INFO:tasks.workunit.client.0.smithi163.stdout: 2 16 100 84 0.164043 0.199219 0.26155 0.335878 2024-05-06T03:48:17.202 INFO:tasks.workunit.client.0.smithi163.stdout: 3 16 162 146 0.190083 0.242188 0.26856 0.314257 2024-05-06T03:48:18.202 INFO:tasks.workunit.client.0.smithi163.stdout: 4 16 226 210 0.205053 0.25 0.251049 0.29621 2024-05-06T03:48:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: 5 16 290 274 0.214035 0.25 0.240187 0.283413 2024-05-06T03:48:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: 6 16 354 338 0.220023 0.25 0.243802 0.279481 2024-05-06T03:48:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: 7 16 418 402 0.224302 0.25 0.240416 0.274279 2024-05-06T03:48:22.203 INFO:tasks.workunit.client.0.smithi163.stdout: 8 16 482 466 0.227509 0.25 0.253872 0.271119 2024-05-06T03:48:23.203 INFO:tasks.workunit.client.0.smithi163.stdout: 9 16 562 546 0.236947 0.3125 0.182293 0.262388 2024-05-06T03:48:24.203 INFO:tasks.workunit.client.0.smithi163.stdout: 10 16 633 617 0.240984 0.277344 0.184518 0.25667 2024-05-06T03:48:25.203 INFO:tasks.workunit.client.0.smithi163.stdout:Total time run: 10.1263 2024-05-06T03:48:25.203 INFO:tasks.workunit.client.0.smithi163.stdout:Total writes made: 633 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Write size: 4096 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Object size: 4096 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Bandwidth (MB/sec): 0.244182 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Stddev Bandwidth: 0.048477 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Max bandwidth (MB/sec): 0.3125 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Min bandwidth (MB/sec): 0.128906 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Average IOPS: 62 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Stddev IOPS: 12.4101 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Max IOPS: 80 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Min IOPS: 33 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Average Latency(s): 0.254871 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Stddev Latency(s): 0.0692647 2024-05-06T03:48:25.204 INFO:tasks.workunit.client.0.smithi163.stdout:Max latency(s): 0.650051 2024-05-06T03:48:25.205 INFO:tasks.workunit.client.0.smithi163.stdout:Min latency(s): 0.149852 2024-05-06T03:48:25.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-05-06T03:48:35.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-05-06T03:48:35.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:48:35.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:48:35.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:48:35.697 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:48:35.697 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T03:48:35.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:48:35.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:48:35.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:48:35.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-05-06T03:48:35.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-05-06T03:48:35.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-05-06T03:48:35.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:48:35.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:48:35.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-05-06T03:48:35.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-05-06T03:48:35.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-05-06T03:48:35.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:48:35.954 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:48:36.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-05-06T03:48:36.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-05-06T03:48:36.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509455' 2024-05-06T03:48:36.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:48:36.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-05-06T03:48:36.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:48:36.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:48:36.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-05-06T03:48:36.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:48:36.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-05-06T03:48:36.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-05-06T03:48:36.070 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836508 2024-05-06T03:48:36.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:48:36.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-05-06T03:48:36.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:48:37.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:48:37.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:48:37.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-05-06T03:48:37.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:48:37.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-05-06T03:48:37.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:48:37.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:48:37.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-05-06T03:48:37.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:48:37.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-05-06T03:48:37.926 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672982 2024-05-06T03:48:37.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-05-06T03:48:37.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:48:38.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-05-06T03:48:38.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:48:38.351 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-05-06T03:48:38.351 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:48:38.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:48:38.352 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-05-06T03:48:38.352 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:48:38.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-05-06T03:48:38.353 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509455 2024-05-06T03:48:38.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-05-06T03:48:38.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:48:38.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-05-06T03:48:38.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-05-06T03:48:38.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-05-06T03:48:39.209 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.376ms); Slow OSD heartbeats on front (longest 0.419ms); 1 pool(s) do not have an application enabled 2024-05-06T03:48:39.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-05-06T03:48:39.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-05-06T03:48:39.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-05-06T03:48:39.749 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.376ms); Slow OSD heartbeats on front (longest 0.419ms); 1 pool(s) do not have an application enabled 2024-05-06T03:48:39.749 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.376ms) 2024-05-06T03:48:39.749 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.376 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.351 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.337 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.324 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.322 msec possibly improving 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.282 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.419ms) 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.419 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.398 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.393 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.373 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.340 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.296 msec 2024-05-06T03:48:39.750 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-05-06T03:48:39.751 INFO:tasks.workunit.client.0.smithi163.stdout: application not enabled on pool 'foo' 2024-05-06T03:48:39.751 INFO:tasks.workunit.client.0.smithi163.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-05-06T03:48:39.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-05-06T03:48:39.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-05-06T03:48:39.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-05-06T03:48:39.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:48:39.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:48:39.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:48:39.875 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:48:39.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:48:39.876 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:48:39.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:48:39.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:48:39.877 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:48:39.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:48:39.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:48:39.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:48:39.879 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:48:39.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:48:39.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:48:39.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-05-06T03:48:39.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:48:39.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:39.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:48:39.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40033 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/network-ping 0 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:48:39.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:48:39.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:48:39.902 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:48:39.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:48:39.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:48:39.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:48:39.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:48:39.904 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:48:39.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:48:39.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:48:39.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:48:39.906 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:48:39.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:48:39.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-06T03:48:39.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-05-06T03:48:39.908 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:48:39.908 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:39.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40033 2024-05-06T03:48:39.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40033 2024-05-06T03:48:39.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:48:39.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:48:39.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-05-06T03:48:39.911 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-06T03:48:39.911 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T03:48:39.936 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-05-06T03:48:39.936 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-05-06T03:48:39.997 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T03:48:40.004 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T03:48:40.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:48:40.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:48:40.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-05-06T03:48:40.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-05-06T03:48:40.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ok-to-stop 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-05-06T03:48:40.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none ' 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none ' 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none ' 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-05-06T03:48:40.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_0_osd 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_1_mon_checks 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_2_mons_checks 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_3_mons_checks 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_4_mons_checks 2024-05-06T03:48:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_5_mons_checks' 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:48:40.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:48:40.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:48:40.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:48:40.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:48:40.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:48:40.013 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:48:40.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:48:40.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:48:40.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:48:40.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:48:40.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:40.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:40.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:48:40.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:48:40.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:48:40.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:48:40.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:48:40.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:40.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:40.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:48:40.017 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:48:40.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:48:40.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:48:40.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:48:40.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:48:40.491 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:48:40.491 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:48:40.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:48:40.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:48:40.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:40.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:40.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:48:40.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:48:40.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:48:40.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:48:40.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:40.776 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:48:40.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:48:40.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-05-06T03:48:40.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:48:41.696 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:48:41.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:48:41.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:48:41.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-05-06T03:48:41.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-05-06T03:48:41.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:48:41.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-05-06T03:48:41.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:48:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1bb7a037-034b-42c1-ba69-2e29f5ee8c1f 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1bb7a037-034b-42c1-ba69-2e29f5ee8c1f' 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 1bb7a037-034b-42c1-ba69-2e29f5ee8c1f 2024-05-06T03:48:41.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:48:41.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZUzhmibzAKxAAPCumdyq0YG6/jny9GGpDbA== 2024-05-06T03:48:41.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZUzhmibzAKxAAPCumdyq0YG6/jny9GGpDbA=="}' 2024-05-06T03:48:41.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1bb7a037-034b-42c1-ba69-2e29f5ee8c1f -i td/ok-to-stop/0/new.json 2024-05-06T03:48:42.016 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:48:42.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-05-06T03:48:42.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZUzhmibzAKxAAPCumdyq0YG6/jny9GGpDbA== --osd-uuid 1bb7a037-034b-42c1-ba69-2e29f5ee8c1f 2024-05-06T03:48:42.053 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.045+0000 7f1c74b87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:48:42.085 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.077+0000 7f1c74b87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:48:42.086 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.078+0000 7f1c74b87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:48:42.086 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.079+0000 7f1c74b87640 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-05-06T03:48:42.086 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.079+0000 7f1c74b87640 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-05-06T03:48:42.087 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.079+0000 7f1c74b87640 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-05-06T03:48:42.125 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:42.117+0000 7f1c74b87640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-05-06T03:48:47.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-05-06T03:48:47.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:48:47.787 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:48:47.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:48:47.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:48:48.334 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:48:48.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:48:48.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:48:48.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:48:48.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:48:48.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:48:48.358 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:48.351+0000 7f8220410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:48:48.373 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:48.366+0000 7f8220410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:48:48.374 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:48.367+0000 7f8220410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:48:48.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:48:48.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:48:48.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:48:48.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:48:48.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:48:48.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:48.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:48:48.780 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:48:48.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:48.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:48.899 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:48.892+0000 7f8220410640 -1 Falling back to public interface 2024-05-06T03:48:49.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:50.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:50.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:50.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:48:50.207 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:48:50.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:50.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:50.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:51.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:48:51.234+0000 7f8220410640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:48:51.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:51.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:51.641 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:48:51.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:48:51.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:51.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:52.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:53.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:53.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:53.171 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:48:53.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:48:53.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:53.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:53.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:54.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:54.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:54.573 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:48:54.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:48:54.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:54.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:55.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:56.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:56.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:56.045 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:48:56.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:48:56.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:56.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:56.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:57.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:57.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:57.445 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:48:57.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:48:57.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:57.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:57.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:48:58.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:48:58.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:48:58.849 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:48:58.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:48:58.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:48:58.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:48:59.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:00.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:00.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:00.310 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:49:00.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:49:00.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:00.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:00.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:01.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:01.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:01.712 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:49:01.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:49:01.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:01.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:02.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:03.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:03.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:03.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:49:03.143 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:49:03.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:03.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:04.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:04.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:04.887 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:49:04.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:49:04.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:04.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:05.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:06.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:06.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:06.418 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:49:06.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:49:06.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:06.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:07.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:08.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:08.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:08.158 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:49:08.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:49:08.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:08.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:09.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:10.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:10.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:10.288 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:49:10.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:49:10.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:10.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:10.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:11.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:11.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:11.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:49:11.747 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:49:11.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:11.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:12.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:13.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:13.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:13.334 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:49:13.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:49:13.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:13.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:13.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:14.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:14.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:14.844 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:49:14.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:49:14.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:14.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:15.273 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1669615316,v1:127.0.0.1:6803/1669615316] [v2:127.0.0.1:6804/1669615316,v1:127.0.0.1:6805/1669615316] exists,up 1bb7a037-034b-42c1-ba69-2e29f5ee8c1f 2024-05-06T03:49:15.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:49:15.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:49:15.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:15.276 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:15.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:49:15.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:49:15.297 INFO:tasks.workunit.client.0.smithi163.stdout:starting mds.a at 2024-05-06T03:49:15.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-05-06T03:49:15.908 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'meta' already exists 2024-05-06T03:49:15.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-05-06T03:49:16.818 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'data' already exists 2024-05-06T03:49:16.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-05-06T03:49:17.780 INFO:tasks.workunit.client.0.smithi163.stderr:filesystem 'myfs' already exists 2024-05-06T03:49:17.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-05-06T03:49:22.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-05-06T03:49:23.149 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-05-06T03:49:23.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-05-06T03:49:23.494 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 2024-05-06T03:49:23.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-05-06T03:49:24.032 INFO:tasks.workunit.client.0.smithi163.stderr:should be safe to stop [dne] 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:49:24.045 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:24.046 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:24.046 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:49:24.046 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:49:24.068 INFO:tasks.workunit.client.0.smithi163.stdout:starting mds.b at 2024-05-06T03:49:24.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-05-06T03:49:29.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-05-06T03:49:29.608 INFO:tasks.workunit.client.0.smithi163.stderr:should be safe to stop [a] 2024-05-06T03:49:29.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-05-06T03:49:30.162 INFO:tasks.workunit.client.0.smithi163.stderr:should be safe to stop [b] 2024-05-06T03:49:30.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-05-06T03:49:30.538 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-05-06T03:49:30.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-05-06T03:49:31.068 INFO:tasks.workunit.client.0.smithi163.stderr:should be safe to stop [a,dne1,dne2] 2024-05-06T03:49:31.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-05-06T03:49:31.608 INFO:tasks.workunit.client.0.smithi163.stderr:should be safe to stop [b,dne] 2024-05-06T03:49:31.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-05-06T03:49:31.984 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4153 and/or standby gids 4167 2024-05-06T03:49:31.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-05-06T03:49:32.518 INFO:tasks.workunit.client.0.smithi163.stderr:should be safe to stop [dne1,dne2] 2024-05-06T03:49:32.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-05-06T03:49:32.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:49:32.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:49:32.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:49:32.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:49:32.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:49:32.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:49:32.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:49:32.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:49:32.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:49:32.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:49:32.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:49:32.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:49:32.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:49:32.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:49:32.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:49:32.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:49:32.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:49:32.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:49:32.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:49:32.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:49:32.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:49:32.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:49:32.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:49:32.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:32.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:32.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:49:32.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:49:32.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:49:32.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:49:32.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:49:32.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:49:32.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:49:32.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:49:32.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:49:32.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:49:32.763 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:49:32.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:49:32.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:49:32.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:49:32.764 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:49:32.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:49:32.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:49:32.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:49:32.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:49:32.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:32.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:32.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:49:32.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:49:32.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:49:32.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:49:32.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:49:32.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:32.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:32.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-05-06T03:49:32.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:49:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:49:33.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:49:33.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:49:33.412 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:49:33.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:49:33.494 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:49:33.495 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:49:33.495 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:33.495 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:33.495 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:49:33.495 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:49:33.495 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:49:33.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-05-06T03:49:33.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-05-06T03:49:33.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:49:33.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:49:33.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:49:33.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-05-06T03:49:33.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:49:34.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:49:34.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:49:34.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:49:34.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-05-06T03:49:34.457 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:49:34.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3807353b-7e37-4521-9cf6-93ccd05fe2d4 2024-05-06T03:49:34.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3807353b-7e37-4521-9cf6-93ccd05fe2d4' 2024-05-06T03:49:34.458 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 3807353b-7e37-4521-9cf6-93ccd05fe2d4 2024-05-06T03:49:34.458 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:49:34.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOUzhm3+e3GxAAmwIVV+MNXHnhUa/qR4OU/w== 2024-05-06T03:49:34.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOUzhm3+e3GxAAmwIVV+MNXHnhUa/qR4OU/w=="}' 2024-05-06T03:49:34.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3807353b-7e37-4521-9cf6-93ccd05fe2d4 -i td/ok-to-stop/0/new.json 2024-05-06T03:49:34.759 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:49:34.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-05-06T03:49:34.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOUzhm3+e3GxAAmwIVV+MNXHnhUa/qR4OU/w== --osd-uuid 3807353b-7e37-4521-9cf6-93ccd05fe2d4 2024-05-06T03:49:34.797 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.790+0000 7f87b1a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:49:34.828 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.821+0000 7f87b1a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:49:34.829 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.822+0000 7f87b1a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:49:34.829 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.822+0000 7f87b1a0c640 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-05-06T03:49:34.829 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.822+0000 7f87b1a0c640 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-05-06T03:49:34.829 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.822+0000 7f87b1a0c640 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-05-06T03:49:34.868 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:34.861+0000 7f87b1a0c640 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-05-06T03:49:40.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-05-06T03:49:40.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:49:40.351 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:49:40.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:49:40.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:49:40.909 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:49:40.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:49:40.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:49:40.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:49:40.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:49:40.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:49:40.933 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:40.926+0000 7f14cb325640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:49:40.950 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:40.943+0000 7f14cb325640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:49:40.951 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:40.944+0000 7f14cb325640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:49:41.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:49:41.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:49:41.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:49:41.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:49:41.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:49:41.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:41.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:49:41.338 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:49:41.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:41.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:41.468 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:41.461+0000 7f14cb325640 -1 Falling back to public interface 2024-05-06T03:49:41.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:42.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:42.756 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:49:42.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:42.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:49:42.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:42.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:43.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:43.800 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:49:43.793+0000 7f14cb325640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:49:44.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:44.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:44.190 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:49:44.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:49:44.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:44.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:44.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:45.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:45.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:45.592 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:49:45.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:49:45.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:45.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:45.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:46.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:46.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:46.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:49:46.997 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:49:46.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:46.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:47.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:48.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:48.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:48.558 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:49:48.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:49:48.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:48.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:48.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:49.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:49.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:49.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:49:49.977 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:49:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:49.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:50.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:51.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:51.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:51.376 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:49:51.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:49:51.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:51.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:51.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:52.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:52.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:52.841 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:49:52.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:49:52.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:52.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:53.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:54.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:54.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:54.256 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:49:54.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:49:54.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:54.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:54.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:55.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:55.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:55.682 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:49:55.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:49:55.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:55.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:56.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:57.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:57.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:57.533 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:49:57.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:49:57.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:57.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:58.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:49:59.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:49:59.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:49:59.141 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:49:59.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:49:59.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:49:59.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:49:59.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:00.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:00.833 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:50:00.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:00.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:50:00.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:00.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:50:01.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:02.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:02.640 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:50:02.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:02.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:50:02.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:02.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:50:03.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:04.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:04.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:04.725 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:50:04.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:50:04.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:04.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:50:05.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:06.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:06.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:06.774 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:50:06.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:50:06.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:06.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:50:07.201 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/876532044,v1:127.0.0.1:6803/876532044] [v2:127.0.0.1:6804/876532044,v1:127.0.0.1:6805/876532044] exists,up 3807353b-7e37-4521-9cf6-93ccd05fe2d4 2024-05-06T03:50:07.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:50:07.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:50:07.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:50:07.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-05-06T03:50:07.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:50:07.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-05-06T03:50:07.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:50:07.208 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 ded5d4ce-11bb-4c99-a21b-46e7550f904f 2024-05-06T03:50:07.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ded5d4ce-11bb-4c99-a21b-46e7550f904f 2024-05-06T03:50:07.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ded5d4ce-11bb-4c99-a21b-46e7550f904f' 2024-05-06T03:50:07.209 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:50:07.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvUzhmaZfGDBAA3ru9VR1qc0gDD/VEX/+e3w== 2024-05-06T03:50:07.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvUzhmaZfGDBAA3ru9VR1qc0gDD/VEX/+e3w=="}' 2024-05-06T03:50:07.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ded5d4ce-11bb-4c99-a21b-46e7550f904f -i td/ok-to-stop/1/new.json 2024-05-06T03:50:07.913 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:50:07.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-05-06T03:50:07.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvUzhmaZfGDBAA3ru9VR1qc0gDD/VEX/+e3w== --osd-uuid ded5d4ce-11bb-4c99-a21b-46e7550f904f 2024-05-06T03:50:07.951 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.945+0000 7f1081c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:07.960 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.953+0000 7f1081c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:07.961 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.954+0000 7f1081c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:07.962 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.954+0000 7f1081c26640 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-05-06T03:50:07.962 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.954+0000 7f1081c26640 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-05-06T03:50:07.962 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.954+0000 7f1081c26640 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-05-06T03:50:07.991 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:07.985+0000 7f1081c26640 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-05-06T03:50:13.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-05-06T03:50:13.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:50:13.730 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T03:50:13.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T03:50:13.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:50:14.284 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T03:50:14.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T03:50:14.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:50:14.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:50:14.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:50:14.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:50:14.308 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:14.301+0000 7fd0ba106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:14.314 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:14.307+0000 7fd0ba106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:14.317 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:14.309+0000 7fd0ba106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:14.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T03:50:14.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:50:14.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T03:50:14.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:50:14.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:50:14.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:14.737 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:50:14.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:50:14.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:14.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:14.849 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:14.843+0000 7fd0ba106640 -1 Falling back to public interface 2024-05-06T03:50:15.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:16.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:16.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:16.180 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:50:16.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:50:16.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:16.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:16.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:17.218 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:17.211+0000 7fd0ba106640 -1 osd.1 0 log_to_monitors true 2024-05-06T03:50:17.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:17.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:17.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:50:17.639 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:50:17.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:17.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:18.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:19.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:19.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:19.040 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:50:19.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:50:19.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:19.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:19.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:20.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:20.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:20.445 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:50:20.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:50:20.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:20.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:20.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:21.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:21.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:21.844 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:50:21.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:50:21.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:21.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:22.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:23.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:23.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:23.279 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:50:23.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:50:23.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:23.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:23.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:24.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:24.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:24.771 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:50:24.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:50:24.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:24.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:25.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:26.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:26.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:26.171 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:50:26.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:50:26.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:26.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:26.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:27.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:27.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:27.581 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:50:27.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:50:27.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:27.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:28.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:29.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:29.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:29.131 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:50:29.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:50:29.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:29.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:29.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:30.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:30.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:30.550 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:50:30.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:50:30.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:30.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:31.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:32.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:32.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:32.114 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:50:32.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:50:32.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:32.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:32.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:33.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:33.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:33.823 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:50:33.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:50:33.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:33.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:34.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:35.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:35.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:35.467 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:50:35.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:50:35.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:35.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:36.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:37.296 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:50:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:50:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:37.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:38.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:38.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:38.955 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:50:38.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:50:38.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:38.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:39.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:40.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:40.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:40.955 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:50:40.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:50:40.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:40.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:50:41.382 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1315064491,v1:127.0.0.1:6811/1315064491] [v2:127.0.0.1:6812/1315064491,v1:127.0.0.1:6813/1315064491] exists,up ded5d4ce-11bb-4c99-a21b-46e7550f904f 2024-05-06T03:50:41.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:50:41.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-05-06T03:50:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:50:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:50:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a4b4e90e-532c-44d5-b69e-59b63d632c74 2024-05-06T03:50:41.390 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 a4b4e90e-532c-44d5-b69e-59b63d632c74 2024-05-06T03:50:41.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a4b4e90e-532c-44d5-b69e-59b63d632c74' 2024-05-06T03:50:41.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:50:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRUzhmHjeIFxAAyU9hRhoodKelC0ym+UzcvA== 2024-05-06T03:50:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRUzhmHjeIFxAAyU9hRhoodKelC0ym+UzcvA=="}' 2024-05-06T03:50:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a4b4e90e-532c-44d5-b69e-59b63d632c74 -i td/ok-to-stop/2/new.json 2024-05-06T03:50:41.858 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:50:41.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-05-06T03:50:41.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRUzhmHjeIFxAAyU9hRhoodKelC0ym+UzcvA== --osd-uuid a4b4e90e-532c-44d5-b69e-59b63d632c74 2024-05-06T03:50:41.897 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:41.890+0000 7f66114a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:41.949 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:41.943+0000 7f66114a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:41.950 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:41.944+0000 7f66114a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:41.951 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:41.944+0000 7f66114a4640 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-05-06T03:50:41.952 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:41.944+0000 7f66114a4640 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-05-06T03:50:41.952 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:41.944+0000 7f66114a4640 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-05-06T03:50:42.048 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:42.042+0000 7f66114a4640 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-05-06T03:50:47.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-05-06T03:50:47.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:50:47.769 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T03:50:47.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T03:50:47.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:50:48.312 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T03:50:48.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T03:50:48.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:50:48.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:50:48.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:50:48.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:50:48.336 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:48.329+0000 7fe63cf1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:48.353 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:48.347+0000 7fe63cf1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:48.354 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:48.348+0000 7fe63cf1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:50:48.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T03:50:48.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:50:48.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T03:50:48.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:50:48.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:50:48.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:48.768 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:50:48.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:50:48.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:48.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:48.897 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:48.891+0000 7fe63cf1c640 -1 Falling back to public interface 2024-05-06T03:50:49.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:50.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:50.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:50.186 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:50:50.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:50:50.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:50.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:50.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:51.238 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:50:51.232+0000 7fe63cf1c640 -1 osd.2 0 log_to_monitors true 2024-05-06T03:50:51.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:51.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:51.636 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:50:51.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:50:51.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:51.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:52.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:53.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:53.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:53.028 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:50:53.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:50:53.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:53.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:53.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:54.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:54.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:54.453 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:50:54.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:50:54.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:54.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:54.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:55.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:55.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:55.863 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:50:55.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:50:55.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:55.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:56.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:57.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:57.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:57.303 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:50:57.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:50:57.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:57.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:57.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:50:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:50:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:50:58.803 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:50:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:50:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:50:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:50:59.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:00.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:00.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:00.336 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:51:00.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:51:00.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:00.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:00.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:01.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:01.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:01.807 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:51:01.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:51:01.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:01.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:02.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:03.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:03.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:03.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:51:03.313 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:51:03.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:03.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:04.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:05.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:05.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:05.243 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:51:05.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:51:05.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:05.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:06.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:07.248 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:51:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:51:07.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:07.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:08.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:09.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:09.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:09.233 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:51:09.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:51:09.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:09.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:10.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:11.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:11.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:11.132 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:51:11.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:51:11.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:11.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:12.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:13.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:13.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:13.253 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:51:13.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:51:13.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:13.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:14.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:15.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:15.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:15.201 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:51:15.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:51:15.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:15.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:51:15.602 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/342768097,v1:127.0.0.1:6819/342768097] [v2:127.0.0.1:6820/342768097,v1:127.0.0.1:6821/342768097] exists,up a4b4e90e-532c-44d5-b69e-59b63d632c74 2024-05-06T03:51:15.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:51:15.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-05-06T03:51:15.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:51:15.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7df766d-41a1-4081-9089-17e12f95cc5e 2024-05-06T03:51:15.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e7df766d-41a1-4081-9089-17e12f95cc5e' 2024-05-06T03:51:15.610 INFO:tasks.workunit.client.0.smithi163.stdout:add osd3 e7df766d-41a1-4081-9089-17e12f95cc5e 2024-05-06T03:51:15.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:51:15.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCzUzhml2a1JBAALFH2PjgAKMW5A2S3/ume4Q== 2024-05-06T03:51:15.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCzUzhml2a1JBAALFH2PjgAKMW5A2S3/ume4Q=="}' 2024-05-06T03:51:15.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7df766d-41a1-4081-9089-17e12f95cc5e -i td/ok-to-stop/3/new.json 2024-05-06T03:51:16.121 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:51:16.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-05-06T03:51:16.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzUzhml2a1JBAALFH2PjgAKMW5A2S3/ume4Q== --osd-uuid e7df766d-41a1-4081-9089-17e12f95cc5e 2024-05-06T03:51:16.159 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.153+0000 7ffab4b87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:51:16.219 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.213+0000 7ffab4b87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:51:16.220 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.214+0000 7ffab4b87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:51:16.220 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.214+0000 7ffab4b87640 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-05-06T03:51:16.220 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.214+0000 7ffab4b87640 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-05-06T03:51:16.220 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.214+0000 7ffab4b87640 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-05-06T03:51:16.325 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:16.320+0000 7ffab4b87640 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-05-06T03:51:21.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-05-06T03:51:21.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:51:21.964 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd3 key to auth repository 2024-05-06T03:51:21.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-06T03:51:21.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:51:22.514 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.3 2024-05-06T03:51:22.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-06T03:51:22.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:51:22.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:51:22.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:51:22.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:51:22.539 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:22.533+0000 7f45b5901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:51:22.548 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:22.542+0000 7f45b5901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:51:22.549 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:22.543+0000 7f45b5901640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:51:22.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-06T03:51:22.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:51:22.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-06T03:51:22.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:51:22.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:51:22.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:22.988 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:51:22.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:51:22.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:22.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:23.080 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:23.075+0000 7f45b5901640 -1 Falling back to public interface 2024-05-06T03:51:23.413 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:24.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:24.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:24.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:51:24.415 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:51:24.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:24.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:24.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:25.434 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:51:25.428+0000 7f45b5901640 -1 osd.3 0 log_to_monitors true 2024-05-06T03:51:25.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:25.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:25.841 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:51:25.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:51:25.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:25.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:26.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:27.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:27.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:27.255 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:51:27.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:51:27.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:27.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:27.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:28.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:28.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:28.779 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:51:28.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:51:28.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:28.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:29.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:30.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:30.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:30.252 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:51:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:51:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:30.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:31.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:31.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:31.656 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:51:31.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:51:31.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:31.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:32.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:33.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:33.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:33.290 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:51:33.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:51:33.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:33.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:33.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:34.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:34.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:34.770 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:51:34.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:51:34.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:34.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:35.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:36.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:36.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:36.244 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:51:36.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:51:36.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:36.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:36.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:37.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:37.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:37.673 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:51:37.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:51:37.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:37.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:38.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:39.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:39.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:39.267 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:51:39.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:51:39.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:39.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:39.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:40.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:40.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:40.891 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:51:40.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:51:40.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:40.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:42.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:43.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:43.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:43.109 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:51:43.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:51:43.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:43.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:43.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:44.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:44.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:44.949 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:51:44.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:51:44.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:44.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:45.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:46.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:46.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:46.693 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:51:46.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:51:46.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:46.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:47.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:51:48.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:51:48.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:51:48.098 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:51:48.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:51:48.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:51:48.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-06T03:51:48.578 INFO:tasks.workunit.client.0.smithi163.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4151827794,v1:127.0.0.1:6827/4151827794] [v2:127.0.0.1:6828/4151827794,v1:127.0.0.1:6829/4151827794] exists,up e7df766d-41a1-4081-9089-17e12f95cc5e 2024-05-06T03:51:48.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:51:48.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:51:48.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:51:48.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-05-06T03:51:49.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-05-06T03:51:49.809 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'ec' already exists 2024-05-06T03:51:49.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-05-06T03:51:49.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T03:51:49.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T03:51:49.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T03:51:49.821 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T03:51:49.822 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T03:51:49.822 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T03:51:49.822 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T03:51:49.822 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T03:51:49.822 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T03:51:49.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T03:51:49.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T03:51:49.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T03:51:49.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T03:51:49.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:51:49.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr:2 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr:3' 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:50.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:51:50.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-05-06T03:51:50.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-05-06T03:51:50.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-05-06T03:51:50.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:50.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:51:51.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-05-06T03:51:51.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-05-06T03:51:51.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672982' 2024-05-06T03:51:51.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:51.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:51:51.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-05-06T03:51:51.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-05-06T03:51:51.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672982 2-64424509453' 2024-05-06T03:51:51.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:51.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-06T03:51:51.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345927 2024-05-06T03:51:51.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345927 2024-05-06T03:51:51.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672982 2-64424509453 3-85899345927' 2024-05-06T03:51:51.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:51.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-05-06T03:51:51.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:51.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:51:51.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-05-06T03:51:51.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:51.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-05-06T03:51:51.448 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836507 2024-05-06T03:51:51.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-05-06T03:51:51.448 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:51:51.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836507 2024-05-06T03:51:51.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:51:52.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:51:52.926 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:51:53.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836507 2024-05-06T03:51:53.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:53.327 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-05-06T03:51:53.327 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:53.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:51:53.328 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-05-06T03:51:53.328 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:53.329 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672982 2024-05-06T03:51:53.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-05-06T03:51:53.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-05-06T03:51:53.329 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:51:53.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-05-06T03:51:53.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:53.723 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-05-06T03:51:53.723 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:53.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:51:53.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-05-06T03:51:53.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:53.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-05-06T03:51:53.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-05-06T03:51:53.726 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509453 2024-05-06T03:51:53.726 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:51:54.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509453 -lt 64424509453 2024-05-06T03:51:54.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:54.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345927 2024-05-06T03:51:54.166 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:54.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-05-06T03:51:54.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345927 2024-05-06T03:51:54.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:54.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345927 2024-05-06T03:51:54.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-05-06T03:51:54.168 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.3 seq 85899345927 2024-05-06T03:51:54.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-06T03:51:54.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345927 -lt 85899345927 2024-05-06T03:51:54.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T03:51:54.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:51:54.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:51:55.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-05-06T03:51:55.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T03:51:55.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T03:51:55.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T03:51:55.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T03:51:55.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T03:51:55.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T03:51:55.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T03:51:55.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=15 2024-05-06T03:51:55.541 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T03:51:55.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:51:55.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:51:56.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 15 = 32 2024-05-06T03:51:56.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 15 '!=' -1 2024-05-06T03:51:56.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-06T03:51:56.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=15 2024-05-06T03:51:56.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T03:51:56.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-06T03:51:56.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T03:51:56.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T03:51:56.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T03:51:56.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T03:51:56.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T03:51:56.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T03:51:56.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T03:51:56.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T03:51:56.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-05-06T03:51:56.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T03:51:56.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:51:56.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:51:57.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-05-06T03:51:57.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T03:51:57.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T03:51:57.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-05-06T03:51:57.719 INFO:tasks.workunit.client.0.smithi163.stderr:set pool 1 min_size to 3 2024-05-06T03:51:57.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-05-06T03:51:57.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T03:51:57.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T03:51:57.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T03:51:57.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T03:51:57.731 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T03:51:57.731 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T03:51:57.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T03:51:57.732 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T03:51:57.732 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T03:51:57.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T03:51:57.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T03:51:57.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T03:51:57.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T03:51:57.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:51:57.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:51:58.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:51:58.206 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:51:58.206 INFO:tasks.workunit.client.0.smithi163.stderr:2 2024-05-06T03:51:58.206 INFO:tasks.workunit.client.0.smithi163.stderr:3' 2024-05-06T03:51:58.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:51:58.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:58.207 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:51:58.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-05-06T03:51:58.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-05-06T03:51:58.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-05-06T03:51:58.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:58.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:51:58.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-05-06T03:51:58.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-05-06T03:51:58.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985' 2024-05-06T03:51:58.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:58.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:51:58.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-05-06T03:51:58.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-05-06T03:51:58.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509456' 2024-05-06T03:51:58.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:51:58.557 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-06T03:51:58.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345930 2024-05-06T03:51:58.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345930 2024-05-06T03:51:58.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509456 3-85899345930' 2024-05-06T03:51:58.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:58.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-05-06T03:51:58.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:58.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:51:58.669 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-05-06T03:51:58.669 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:58.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-05-06T03:51:58.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-05-06T03:51:58.670 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836511 2024-05-06T03:51:58.670 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:51:59.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-05-06T03:51:59.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:59.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-05-06T03:51:59.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:59.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:51:59.097 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-05-06T03:51:59.097 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:59.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-05-06T03:51:59.098 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672985 2024-05-06T03:51:59.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-05-06T03:51:59.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:51:59.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-05-06T03:51:59.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:51:59.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-05-06T03:51:59.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:51:59.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:51:59.523 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-05-06T03:51:59.523 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:51:59.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-05-06T03:51:59.524 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509456 2024-05-06T03:51:59.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-05-06T03:51:59.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:51:59.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509456 2024-05-06T03:51:59.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:52:00.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:52:00.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:52:01.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509456 2024-05-06T03:52:01.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:01.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345930 2024-05-06T03:52:01.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:01.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-05-06T03:52:01.392 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345930 2024-05-06T03:52:01.392 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:01.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345930 2024-05-06T03:52:01.393 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.3 seq 85899345930 2024-05-06T03:52:01.394 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-05-06T03:52:01.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-06T03:52:01.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345931 -lt 85899345930 2024-05-06T03:52:01.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T03:52:01.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:01.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:02.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-05-06T03:52:02.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T03:52:02.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T03:52:02.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T03:52:02.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T03:52:02.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T03:52:02.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T03:52:02.387 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T03:52:02.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-05-06T03:52:02.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T03:52:02.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:02.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:03.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-05-06T03:52:03.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T03:52:03.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T03:52:03.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-05-06T03:52:03.730 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:03.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-05-06T03:52:04.116 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:04.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-05-06T03:52:04.499 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:04.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-05-06T03:52:04.950 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:04.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-05-06T03:52:05.226 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:05.227 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-05-06T03:52:05.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-05-06T03:52:05.497 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:05.497 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-05-06T03:52:05.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-05-06T03:52:05.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-05-06T03:52:05.930 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:05.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-05-06T03:52:05.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-05-06T03:52:06.319 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:06.320 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-05-06T03:52:07.264 INFO:tasks.workunit.client.0.smithi163.stderr:set pool 1 min_size to 2 2024-05-06T03:52:07.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-05-06T03:52:07.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T03:52:07.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T03:52:07.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T03:52:07.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T03:52:07.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T03:52:07.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T03:52:07.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T03:52:07.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T03:52:07.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T03:52:07.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T03:52:07.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T03:52:07.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T03:52:07.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T03:52:07.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:52:07.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr:2 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr:3' 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:07.980 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:52:08.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836515 2024-05-06T03:52:08.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836515 2024-05-06T03:52:08.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515' 2024-05-06T03:52:08.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:08.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:52:08.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-05-06T03:52:08.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-05-06T03:52:08.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989' 2024-05-06T03:52:08.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:08.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:52:08.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-05-06T03:52:08.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-05-06T03:52:08.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509460' 2024-05-06T03:52:08.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:08.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-06T03:52:08.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345934 2024-05-06T03:52:08.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345934 2024-05-06T03:52:08.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509460 3-85899345934' 2024-05-06T03:52:08.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:08.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836515 2024-05-06T03:52:08.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:08.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:52:08.550 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-05-06T03:52:08.550 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:08.551 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836515 2024-05-06T03:52:08.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836515 2024-05-06T03:52:08.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-05-06T03:52:08.551 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:52:09.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836515 -lt 21474836515 2024-05-06T03:52:09.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:09.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-05-06T03:52:09.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:09.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:52:09.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-05-06T03:52:09.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:09.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-05-06T03:52:09.016 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672989 2024-05-06T03:52:09.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-05-06T03:52:09.016 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:52:09.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-05-06T03:52:09.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:09.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-05-06T03:52:09.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:09.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:52:09.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-05-06T03:52:09.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:09.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-05-06T03:52:09.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-05-06T03:52:09.442 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509460 2024-05-06T03:52:09.442 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:52:09.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-05-06T03:52:09.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:09.849 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345934 2024-05-06T03:52:09.849 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:09.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-05-06T03:52:09.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345934 2024-05-06T03:52:09.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:09.851 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.3 seq 85899345934 2024-05-06T03:52:09.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345934 2024-05-06T03:52:09.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-05-06T03:52:09.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-06T03:52:10.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345934 -lt 85899345934 2024-05-06T03:52:10.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T03:52:10.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:10.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T03:52:10.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T03:52:11.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-05-06T03:52:11.234 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T03:52:11.234 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:11.234 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:11.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-05-06T03:52:11.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' -1 2024-05-06T03:52:11.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-06T03:52:11.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=11 2024-05-06T03:52:11.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T03:52:11.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-06T03:52:11.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T03:52:11.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T03:52:11.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T03:52:11.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T03:52:11.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T03:52:11.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T03:52:11.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T03:52:11.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T03:52:12.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=11 2024-05-06T03:52:12.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T03:52:12.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:12.276 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 11 = 32 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 11 '!=' 11 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T03:52:12.847 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T03:52:13.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T03:52:13.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T03:52:13.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-05-06T03:52:13.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T03:52:13.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-05-06T03:52:13.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T03:52:13.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T03:52:13.583 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T03:52:13.583 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T03:52:13.583 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T03:52:13.583 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T03:52:13.584 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T03:52:13.584 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T03:52:13.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-05-06T03:52:13.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T03:52:13.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:13.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:14.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-05-06T03:52:14.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T03:52:14.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T03:52:14.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-05-06T03:52:14.926 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:14.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-05-06T03:52:15.314 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:15.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-05-06T03:52:15.597 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:15.597 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-05-06T03:52:15.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-05-06T03:52:15.870 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:15.870 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-05-06T03:52:15.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-05-06T03:52:15.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-05-06T03:52:16.261 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:16.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-05-06T03:52:16.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-05-06T03:52:16.652 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:16.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-05-06T03:52:16.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-05-06T03:52:17.037 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:17.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-05-06T03:52:17.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-05-06T03:52:17.424 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:17.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-05-06T03:52:17.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:17.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:17.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:17.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:17.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:17.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:17.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-05-06T03:52:18.569 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already down. 2024-05-06T03:52:18.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-05-06T03:52:18.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-05-06T03:52:18.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-05-06T03:52:18.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-05-06T03:52:18.581 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-05-06T03:52:18.581 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T03:52:18.581 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T03:52:18.582 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T03:52:18.582 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T03:52:18.582 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T03:52:18.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T03:52:18.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-05-06T03:52:18.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-05-06T03:52:18.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-05-06T03:52:18.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:52:18.664 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:52:19.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T03:52:19.288 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T03:52:19.288 INFO:tasks.workunit.client.0.smithi163.stderr:2 2024-05-06T03:52:19.288 INFO:tasks.workunit.client.0.smithi163.stderr:3' 2024-05-06T03:52:19.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:52:19.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:19.288 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:52:19.364 INFO:tasks.workunit.client.0.smithi163.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-05-06T03:52:19.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-05-06T03:52:19.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-05-06T03:52:19.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-05-06T03:52:19.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:19.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T03:52:19.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672993 2024-05-06T03:52:19.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672993 2024-05-06T03:52:19.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672993' 2024-05-06T03:52:19.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:19.480 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T03:52:19.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509464 2024-05-06T03:52:19.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509464 2024-05-06T03:52:19.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672993 2-64424509464' 2024-05-06T03:52:19.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:52:19.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-06T03:52:19.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345939 2024-05-06T03:52:19.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345939 2024-05-06T03:52:19.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672993 2-64424509464 3-85899345939' 2024-05-06T03:52:19.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:19.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672993 2024-05-06T03:52:19.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:19.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T03:52:19.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672993 2024-05-06T03:52:19.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:19.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672993 2024-05-06T03:52:19.729 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672993 2024-05-06T03:52:19.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-05-06T03:52:19.730 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:52:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672991 -lt 42949672993 2024-05-06T03:52:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:52:21.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:52:21.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T03:52:21.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672993 -lt 42949672993 2024-05-06T03:52:21.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:21.583 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509464 2024-05-06T03:52:21.583 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:21.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T03:52:21.584 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509464 2024-05-06T03:52:21.584 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:21.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509464 2024-05-06T03:52:21.585 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509464 2024-05-06T03:52:21.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-05-06T03:52:21.585 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T03:52:22.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509465 -lt 64424509464 2024-05-06T03:52:22.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:52:22.017 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345939 2024-05-06T03:52:22.017 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:52:22.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-05-06T03:52:22.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345939 2024-05-06T03:52:22.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:52:22.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345939 2024-05-06T03:52:22.020 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.3 seq 85899345939 2024-05-06T03:52:22.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-05-06T03:52:22.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-06T03:52:22.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345939 -lt 85899345939 2024-05-06T03:52:22.443 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-05-06T03:52:22.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:22.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-05-06T03:52:22.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-05-06T03:52:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-05-06T03:52:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-05-06T03:52:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T03:52:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T03:52:23.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-05-06T03:52:23.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-05-06T03:52:23.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-05-06T03:52:23.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-05-06T03:52:24.274 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-05-06T03:52:24.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-05-06T03:52:24.661 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:24.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-05-06T03:52:24.938 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:24.938 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-05-06T03:52:24.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-05-06T03:52:25.210 INFO:tasks.workunit.client.0.smithi163.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-05-06T03:52:25.210 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-05-06T03:52:25.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:25.214 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:25.215 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:25.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:25.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:25.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:25.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:25.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:52:25.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:52:25.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:52:25.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:25.324 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:25.324 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:25.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:25.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:25.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:25.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:25.329 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:52:25.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:52:25.330 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:52:25.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:52:25.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:52:25.331 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:52:25.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:25.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:52:25.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:52:25.333 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:25.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:52:25.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:52:25.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:52:25.355 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:52:25.356 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:25.356 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:25.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:52:25.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:25.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:25.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:25.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:25.359 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:52:25.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:52:25.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:52:25.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:52:25.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:52:25.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:52:25.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:25.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:52:25.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:52:25.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:25.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:52:25.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:52:25.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:52:25.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:52:25.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:25.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:25.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:52:25.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:52:25.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:52:25.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:52:25.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:52:25.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:25.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:25.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:52:25.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:52:25.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:52:25.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:52:25.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:52:26.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:52:26.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:52:26.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:26.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:26.102 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:52:26.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:26.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:52:26.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-05-06T03:52:26.966 INFO:tasks.workunit.client.0.smithi163.stderr:invalid mon(s) specified: [dne] 2024-05-06T03:52:26.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-05-06T03:52:27.134 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-05-06T03:52:27.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-05-06T03:52:27.284 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-05-06T03:52:27.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-05-06T03:52:27.435 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: removing mon.a would break quorum 2024-05-06T03:52:27.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-05-06T03:52:27.626 INFO:tasks.workunit.client.0.smithi163.stderr:mon.dne does not exist 2024-05-06T03:52:27.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:27.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:27.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:27.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:27.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:27.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:27.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:27.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:27.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:27.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:52:27.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:52:27.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:52:27.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:52:27.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:52:27.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:52:27.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:27.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:52:27.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:52:27.748 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:27.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:52:27.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:52:27.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:52:27.751 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:52:27.751 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:27.751 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:27.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:52:27.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:27.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:27.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:52:27.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:52:27.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:52:27.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:52:27.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:52:27.756 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:52:27.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:27.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:52:27.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:52:27.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:27.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:52:27.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:52:27.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:52:27.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:52:27.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:27.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:27.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:52:27.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:52:27.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:52:27.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:52:27.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:52:27.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:27.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:27.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:52:27.764 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:52:27.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:52:28.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:52:28.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:52:28.420 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:52:28.421 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:52:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:52:28.422 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:28.422 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:52:28.505 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:52:28.505 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:28.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-05-06T03:52:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:52:29.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-05-06T03:52:29.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:52:29.216 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:29.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get fsid 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:52:29.297 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:29.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get mon_host 2024-05-06T03:52:29.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-05-06T03:52:35.562 INFO:tasks.workunit.client.0.smithi163.stderr:invalid mon(s) specified: [dne] 2024-05-06T03:52:35.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:52:35.729 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-05-06T03:52:35.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:52:35.929 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-05-06T03:52:35.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-05-06T03:52:36.131 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-05-06T03:52:36.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:52:36.390 INFO:tasks.workunit.client.0.smithi163.stderr:adding another mon that is not yet online will not break quorum 2024-05-06T03:52:36.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:52:36.608 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:52:36.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:52:36.902 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:52:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-05-06T03:52:37.162 INFO:tasks.workunit.client.0.smithi163.stderr:mon.dne does not exist 2024-05-06T03:52:37.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:37.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:37.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:37.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:37.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:37.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:37.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:37.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:37.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:52:37.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:52:37.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:52:37.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:52:37.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:52:37.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:52:37.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:37.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:52:37.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:52:37.285 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:37.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:52:37.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:52:37.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:52:37.288 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:52:37.288 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:37.288 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:37.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:52:37.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:37.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:52:37.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:52:37.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:52:37.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:52:37.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:52:37.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:52:37.300 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:52:37.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:37.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:37.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:52:37.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:52:37.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:52:37.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:52:37.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:52:37.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:52:37.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:52:37.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:52:37.746 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:37.746 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:37.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:52:37.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:52:37.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:52:37.914 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:52:37.914 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:37.914 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:37.915 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:52:37.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:52:37.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:37.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:52:37.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:52:37.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:37.996 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-05-06T03:52:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-05-06T03:52:38.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:52:38.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-05-06T03:52:38.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:52:38.719 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:38.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get fsid 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:38.799 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:38.800 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:52:38.800 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:38.800 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get mon_host 2024-05-06T03:52:38.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-05-06T03:52:38.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:52:38.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:52:38.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-05-06T03:52:38.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:52:38.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-05-06T03:52:38.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:39.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:52:39.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-05-06T03:52:39.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:39.527 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:39.527 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.c.asok 2024-05-06T03:52:39.527 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:39.527 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.c.asok config get fsid 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:52:39.601 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:52:39.602 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:52:39.602 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.c.asok 2024-05-06T03:52:39.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:52:39.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.c.asok config get mon_host 2024-05-06T03:52:39.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:52:39.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967619 2024-05-06T03:52:39.682 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:39.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967559 -lt 1714967619 ]] 2024-05-06T03:52:39.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:52:39.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:42.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:42.878 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:42.878 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:52:42.878 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:52:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:52:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:52:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:52:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:52:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-05-06T03:52:43.145 INFO:tasks.workunit.client.0.smithi163.stderr:invalid mon(s) specified: [dne] 2024-05-06T03:52:43.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:52:43.414 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c) after stopping [a] 2024-05-06T03:52:43.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:52:43.665 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,c) after stopping [b] 2024-05-06T03:52:43.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:52:43.925 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b) after stopping [c] 2024-05-06T03:52:43.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-05-06T03:52:44.135 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-05-06T03:52:44.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-05-06T03:52:44.387 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-05-06T03:52:44.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-05-06T03:52:44.638 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-05-06T03:52:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:52:44.940 INFO:tasks.workunit.client.0.smithi163.stderr:adding another mon that is not yet online will not break quorum 2024-05-06T03:52:44.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:52:45.217 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:52:45.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:52:45.476 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:52:45.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:52:45.736 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.c 2024-05-06T03:52:45.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-05-06T03:52:45.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:52:45.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:52:45.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:52:45.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:52:45.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:52:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:52:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-05-06T03:52:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:52:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-05-06T03:52:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:52:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-05-06T03:52:45.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:52:45.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:52:45.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967625 2024-05-06T03:52:45.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:45.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967565 -lt 1714967625 ]] 2024-05-06T03:52:45.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:45.854 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:46.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:46.045 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.046 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.046 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:46.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:46.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:46.048 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:46.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967566 -lt 1714967625 ]] 2024-05-06T03:52:46.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:46.049 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:46.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:46.241 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.242 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.242 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:46.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:46.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:46.244 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:46.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967566 -lt 1714967625 ]] 2024-05-06T03:52:46.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:46.245 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:46.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:46.436 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:46.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:46.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:46.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:46.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967566 -lt 1714967625 ]] 2024-05-06T03:52:46.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:46.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:46.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:46.631 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:46.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:46.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:46.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:46.634 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:46.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967566 -lt 1714967625 ]] 2024-05-06T03:52:46.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:46.635 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:49.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:49.832 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:49.832 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:49.832 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:49.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:49.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:49.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:49.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967569 -lt 1714967625 ]] 2024-05-06T03:52:49.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:49.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:50.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:50.028 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:50.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:50.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:50.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:50.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:50.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:50.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967570 -lt 1714967625 ]] 2024-05-06T03:52:50.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:50.032 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:53.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:53.228 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.229 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.229 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:53.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:53.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:53.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:53.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967573 -lt 1714967625 ]] 2024-05-06T03:52:53.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:53.233 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:53.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:53.423 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.424 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.424 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:53.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:53.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:53.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:53.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967573 -lt 1714967625 ]] 2024-05-06T03:52:53.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:53.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:53.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:53.618 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.618 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:53.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:53.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:53.621 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:53.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967573 -lt 1714967625 ]] 2024-05-06T03:52:53.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:53.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:53.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:53.811 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.811 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:53.812 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:53.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:53.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:53.814 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:53.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967573 -lt 1714967625 ]] 2024-05-06T03:52:53.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:53.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:54.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:54.005 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:54.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:54.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:54.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:54.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:54.008 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:54.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967574 -lt 1714967625 ]] 2024-05-06T03:52:54.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:54.009 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:52:54.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:52:54.199 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:54.199 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:52:54.199 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:52:54.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:52:54.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:52:54.202 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:52:54.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967574 -lt 1714967625 ]] 2024-05-06T03:52:54.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-05-06T03:52:54.203 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:01.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:01.089 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:01.089 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:52:37.316855Z","created":"2024-05-06T03:52:37.316855Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:01.089 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-05-06T03:53:01.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:53:01.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:53:01.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:53:01.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:53:01.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:53:01.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:53:01.347 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-05-06T03:53:01.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:53:01.729 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,c) after stopping [b] 2024-05-06T03:53:01.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:53:04.885 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-05-06T03:53:04.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:53:05.151 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-05-06T03:53:05.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:53:05.403 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: removing mon.a would break quorum 2024-05-06T03:53:05.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:53:05.677 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:53:05.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:53:08.833 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: removing mon.c would break quorum 2024-05-06T03:53:08.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:53:08.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:08.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:08.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:08.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:08.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:08.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:08.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:53:08.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:08.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:08.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:08.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:53:08.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:53:08.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:53:08.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:53:08.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:53:08.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:53:08.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:08.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:53:08.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:53:08.951 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:08.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:53:08.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:53:08.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:53:08.957 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:53:08.957 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:08.957 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:08.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:53:08.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:53:08.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:53:08.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:53:08.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:08.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:08.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:08.961 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:53:08.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:53:08.962 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:53:08.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:53:08.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:53:08.964 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:53:08.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:08.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:53:08.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:53:08.965 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:08.965 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:53:08.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:53:08.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:53:08.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:53:08.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:08.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:08.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:53:08.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:53:08.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:53:08.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:53:08.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:53:08.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:08.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:08.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:53:08.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:53:08.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:53:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:09.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:09.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:53:09.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:53:09.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:53:09.632 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:09.632 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:09.632 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:09.632 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:53:09.632 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:09.632 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:09.717 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:53:09.718 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:09.718 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:53:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-05-06T03:53:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T03:53:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-05-06T03:53:09.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-05-06T03:53:10.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:10.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:10.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:10.241 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:10.241 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:10.242 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:10.242 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:10.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-05-06T03:53:10.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:10.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T03:53:10.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:10.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:53:10.427 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:10.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:53:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:53:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:53:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:10.429 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:10.429 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:10.429 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:53:10.429 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:10.429 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get fsid 2024-05-06T03:53:10.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T03:53:10.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:10.508 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:10.509 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:53:10.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:10.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get mon_host 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-05-06T03:53:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-05-06T03:53:11.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:11.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:11.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:11.012 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:11.013 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:11.013 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:11.013 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:11.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-05-06T03:53:11.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:11.199 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-05-06T03:53:11.199 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:11.199 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.c.asok 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:11.200 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.c.asok config get fsid 2024-05-06T03:53:11.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-05-06T03:53:11.282 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-05-06T03:53:11.283 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:11.283 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:11.283 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:11.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.c.asok 2024-05-06T03:53:11.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:11.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.c.asok config get mon_host 2024-05-06T03:53:11.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-05-06T03:53:11.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:11.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:11.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-05-06T03:53:11.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:11.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-05-06T03:53:11.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-05-06T03:53:11.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:11.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-05-06T03:53:12.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:12.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-05-06T03:53:12.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:12.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-05-06T03:53:12.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:12.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.d.asok 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:12.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.d.asok config get fsid 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-05-06T03:53:12.099 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-05-06T03:53:12.100 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:12.100 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:12.100 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:12.100 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.d.asok 2024-05-06T03:53:12.100 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:12.100 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.d.asok config get mon_host 2024-05-06T03:53:12.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-05-06T03:53:12.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:53:12.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-05-06T03:53:12.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:53:12.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-05-06T03:53:12.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:53:12.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:53:12.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967652 2024-05-06T03:53:12.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:12.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967592 -lt 1714967652 ]] 2024-05-06T03:53:12.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:53:12.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:15.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:15.376 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:15.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:53:15.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:15.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:53:15.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:53:15.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:53:15.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:53:15.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:53:15.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-05-06T03:53:15.644 INFO:tasks.workunit.client.0.smithi163.stderr:invalid mon(s) specified: [dne] 2024-05-06T03:53:15.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:53:15.937 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-05-06T03:53:15.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:53:16.165 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-05-06T03:53:16.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:53:16.635 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-05-06T03:53:16.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-05-06T03:53:16.937 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-05-06T03:53:16.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-05-06T03:53:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-05-06T03:53:17.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-05-06T03:53:17.313 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-05-06T03:53:17.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:53:17.568 INFO:tasks.workunit.client.0.smithi163.stderr:adding another mon that is not yet online will not break quorum 2024-05-06T03:53:17.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:53:17.871 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:53:17.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:53:18.180 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:53:18.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:53:18.474 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.c 2024-05-06T03:53:18.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-05-06T03:53:18.484 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:18.484 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:18.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:18.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:18.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:18.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:18.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-05-06T03:53:18.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:53:18.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-05-06T03:53:18.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:53:18.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-05-06T03:53:18.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:53:18.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:53:18.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967658 2024-05-06T03:53:18.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:18.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967598 -lt 1714967658 ]] 2024-05-06T03:53:18.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:18.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:18.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:18.783 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:18.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:18.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:18.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:18.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:18.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:18.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967598 -lt 1714967658 ]] 2024-05-06T03:53:18.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:18.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:18.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:18.981 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:18.982 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:18.982 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:18.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:18.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:18.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:18.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967598 -lt 1714967658 ]] 2024-05-06T03:53:18.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:18.985 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:19.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:19.178 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:19.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:19.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:19.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:19.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:19.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:19.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967599 -lt 1714967658 ]] 2024-05-06T03:53:19.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:19.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:19.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:19.371 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:19.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:19.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:19.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:19.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:19.374 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:19.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967599 -lt 1714967658 ]] 2024-05-06T03:53:19.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:19.375 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:19.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:19.564 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:19.564 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:19.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:19.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:19.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:19.567 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:19.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967599 -lt 1714967658 ]] 2024-05-06T03:53:19.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:19.568 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:22.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:22.763 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:22.764 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:22.764 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:22.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:22.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:22.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:22.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967602 -lt 1714967658 ]] 2024-05-06T03:53:22.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:22.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:25.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:25.966 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:25.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:25.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:25.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:53:25.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:53:25.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:25.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967605 -lt 1714967658 ]] 2024-05-06T03:53:25.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:53:25.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:38.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:38.818 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:38.818 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:08.988083Z","created":"2024-05-06T03:53:08.988083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:38.818 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:53:38.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:53:38.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:53:38.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:53:38.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:53:38.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:53:38.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:53:39.218 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-05-06T03:53:39.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:53:39.470 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-05-06T03:53:39.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:53:39.686 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-05-06T03:53:39.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-05-06T03:53:39.907 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-05-06T03:53:39.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:53:40.215 INFO:tasks.workunit.client.0.smithi163.stderr:adding another mon that is not yet online will not break quorum 2024-05-06T03:53:40.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:53:40.477 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:53:40.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:53:40.744 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:53:40.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:53:40.987 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.c 2024-05-06T03:53:40.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-05-06T03:53:44.190 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.d 2024-05-06T03:53:44.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:53:44.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:44.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:44.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:44.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:44.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:44.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:44.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:44.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:44.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:44.312 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:53:44.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:53:44.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:53:44.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:53:44.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:53:44.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:53:44.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:44.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:53:44.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:53:44.316 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:44.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:53:44.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:53:44.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:53:44.322 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:53:44.322 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:44.323 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:44.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:53:44.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:53:44.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:53:44.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-05-06T03:53:44.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:53:44.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:53:44.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:53:44.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:53:44.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:53:44.327 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:53:44.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:53:44.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:53:44.328 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:53:44.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:44.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:53:44.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:53:44.329 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:53:44.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:53:44.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:53:44.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:53:44.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:53:44.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:44.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:44.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:53:44.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:53:44.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:53:44.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-05-06T03:53:44.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:53:44.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:44.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:44.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52628 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:44.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-05-06T03:53:44.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-05-06T03:53:44.797 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:44.797 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:44.797 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:44.797 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:44.797 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:44.797 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:44.798 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:44.798 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-05-06T03:53:44.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:53:45.000 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:45.001 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:45.001 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:45.001 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:53:45.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:45.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get fsid 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.a.asok 2024-05-06T03:53:45.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:45.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.a.asok config get mon_host 2024-05-06T03:53:45.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-05-06T03:53:45.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:45.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:45.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T03:53:45.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:45.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-05-06T03:53:45.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-05-06T03:53:45.609 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:45.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-05-06T03:53:45.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:45.846 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:45.847 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:53:45.847 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:45.847 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get fsid 2024-05-06T03:53:45.926 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.b.asok 2024-05-06T03:53:45.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:45.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.b.asok config get mon_host 2024-05-06T03:53:46.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-05-06T03:53:46.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:46.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:46.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-05-06T03:53:46.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:46.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-05-06T03:53:46.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:46.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-05-06T03:53:46.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:46.668 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:46.669 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:46.669 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.c.asok 2024-05-06T03:53:46.669 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:46.669 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.c.asok config get fsid 2024-05-06T03:53:46.751 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:46.752 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:46.753 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:46.753 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.c.asok 2024-05-06T03:53:46.753 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:46.753 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.c.asok config get mon_host 2024-05-06T03:53:46.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-05-06T03:53:46.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:46.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:46.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-05-06T03:53:46.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:46.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-05-06T03:53:46.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-05-06T03:53:47.285 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:47.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-05-06T03:53:47.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-05-06T03:53:47.464 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:47.465 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:47.465 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:47.465 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.d.asok 2024-05-06T03:53:47.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:47.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.d.asok config get fsid 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:47.544 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:47.545 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.d.asok 2024-05-06T03:53:47.545 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:47.545 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.d.asok config get mon_host 2024-05-06T03:53:47.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-05-06T03:53:47.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-05-06T03:53:47.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:53:47.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-05-06T03:53:47.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:53:47.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-05-06T03:53:47.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52628/$cluster-$name.asok' 2024-05-06T03:53:48.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.52628/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-05-06T03:53:48.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:48.277 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:48.278 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.e.asok 2024-05-06T03:53:48.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:48.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.e.asok config get fsid 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:53:48.357 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:53:48.358 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:53:48.358 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52628/ceph-mon.e.asok 2024-05-06T03:53:48.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:53:48.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.52628/ceph-mon.e.asok config get mon_host 2024-05-06T03:53:48.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-05-06T03:53:48.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:53:48.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-05-06T03:53:48.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:53:48.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-05-06T03:53:48.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:53:48.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:53:48.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967688 2024-05-06T03:53:48.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:53:48.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967628 -lt 1714967688 ]] 2024-05-06T03:53:48.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-05-06T03:53:48.438 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:53:53.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:53:53.161 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:53.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-05-06T03:53:53.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:53:53.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:53:53.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:53:53.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:53:53.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:53:53.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:53:53.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-05-06T03:53:53.437 INFO:tasks.workunit.client.0.smithi163.stderr:invalid mon(s) specified: [dne] 2024-05-06T03:53:53.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:53:53.715 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-05-06T03:53:53.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:53:53.984 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-05-06T03:53:53.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:53:54.268 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-05-06T03:53:54.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-05-06T03:53:54.513 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-05-06T03:53:54.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-05-06T03:53:57.710 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-05-06T03:53:57.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-05-06T03:53:57.991 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-05-06T03:53:58.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-05-06T03:54:01.191 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-05-06T03:54:01.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-05-06T03:54:04.352 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-05-06T03:54:04.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:54:04.631 INFO:tasks.workunit.client.0.smithi163.stderr:adding another mon that is not yet online will not break quorum 2024-05-06T03:54:04.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:54:05.142 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:54:05.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:54:05.386 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:54:05.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:54:05.663 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.c 2024-05-06T03:54:05.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-05-06T03:54:05.958 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.d 2024-05-06T03:54:05.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-05-06T03:54:06.284 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.e 2024-05-06T03:54:06.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-05-06T03:54:06.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:54:06.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:54:06.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:54:06.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:54:06.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:54:06.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:54:06.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967706 2024-05-06T03:54:06.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:06.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967646 -lt 1714967706 ]] 2024-05-06T03:54:06.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:06.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:06.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:06.592 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:06.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:06.593 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:06.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:06.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:06.595 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:06.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967646 -lt 1714967706 ]] 2024-05-06T03:54:06.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:06.596 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:06.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:06.788 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:06.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:06.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:06.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:06.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:06.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:06.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967646 -lt 1714967706 ]] 2024-05-06T03:54:06.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:06.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:06.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:06.984 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:06.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:06.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:06.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:06.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:06.987 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:06.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967646 -lt 1714967706 ]] 2024-05-06T03:54:06.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:06.988 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:07.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:07.181 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:07.182 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:07.182 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:07.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:07.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:07.185 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:07.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967647 -lt 1714967706 ]] 2024-05-06T03:54:07.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:07.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:10.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:10.382 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:10.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:10.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:10.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:10.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:10.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:10.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967650 -lt 1714967706 ]] 2024-05-06T03:54:10.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:10.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:10.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:10.578 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:10.578 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:10.578 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:10.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:10.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:10.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:10.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967650 -lt 1714967706 ]] 2024-05-06T03:54:10.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:10.581 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:13.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:13.778 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:13.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:13.779 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:13.781 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:13.781 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:13.781 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:13.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967653 -lt 1714967706 ]] 2024-05-06T03:54:13.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:13.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:13.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:13.974 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:13.974 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:13.974 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:13.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:13.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:13.977 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:13.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967653 -lt 1714967706 ]] 2024-05-06T03:54:13.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:13.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:14.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:14.166 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:14.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:14.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:14.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:14.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967654 -lt 1714967706 ]] 2024-05-06T03:54:14.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:14.170 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:14.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:14.359 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.359 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:14.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:14.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:14.362 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:14.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967654 -lt 1714967706 ]] 2024-05-06T03:54:14.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:14.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:14.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:14.554 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.554 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.554 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:14.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:14.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:14.557 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:14.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967654 -lt 1714967706 ]] 2024-05-06T03:54:14.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:14.558 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:14.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:14.748 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.748 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.748 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:14.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:14.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:14.751 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:14.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967654 -lt 1714967706 ]] 2024-05-06T03:54:14.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:14.752 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:14.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:14.943 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.943 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:14.943 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:14.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:14.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:14.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:14.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967654 -lt 1714967706 ]] 2024-05-06T03:54:14.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-05-06T03:54:14.946 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:21.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:21.851 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:21.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-05-06T03:54:21.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:21.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:54:21.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:54:21.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:54:21.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:54:21.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:54:21.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:54:22.082 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-05-06T03:54:22.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:54:25.284 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-05-06T03:54:25.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:54:25.559 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-05-06T03:54:25.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-05-06T03:54:25.819 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-05-06T03:54:25.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-05-06T03:54:26.089 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-05-06T03:54:26.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:54:26.381 INFO:tasks.workunit.client.0.smithi163.stderr:adding another mon that is not yet online will not break quorum 2024-05-06T03:54:26.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:54:26.667 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:54:26.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:54:26.952 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.b 2024-05-06T03:54:26.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:54:27.336 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.c 2024-05-06T03:54:27.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-05-06T03:54:27.639 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.d 2024-05-06T03:54:27.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-05-06T03:54:27.891 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.e 2024-05-06T03:54:27.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-05-06T03:54:27.902 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:54:27.902 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:54:27.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:54:27.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:54:27.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:54:28.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:54:28.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-05-06T03:54:28.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-05-06T03:54:28.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-05-06T03:54:28.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-05-06T03:54:28.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-05-06T03:54:28.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-05-06T03:54:28.008 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-05-06T03:54:28.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1714967728 2024-05-06T03:54:28.008 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:28.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967668 -lt 1714967728 ]] 2024-05-06T03:54:28.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:28.009 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:28.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:28.201 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:28.202 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:28.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:28.204 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:28.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967668 -lt 1714967728 ]] 2024-05-06T03:54:28.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:28.205 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:28.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:28.398 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:28.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:28.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:28.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:28.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967668 -lt 1714967728 ]] 2024-05-06T03:54:28.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:28.403 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:28.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:28.595 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.595 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:28.595 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:28.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:28.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:28.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967668 -lt 1714967728 ]] 2024-05-06T03:54:28.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:28.599 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:28.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:28.790 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:28.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:28.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:28.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:28.793 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:28.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967668 -lt 1714967728 ]] 2024-05-06T03:54:28.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:28.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:31.989 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:31.990 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:31.990 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:31.990 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:31.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:31.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:31.993 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:31.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967671 -lt 1714967728 ]] 2024-05-06T03:54:31.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:31.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:32.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:32.187 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.188 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.188 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:32.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:32.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:32.190 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:32.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967672 -lt 1714967728 ]] 2024-05-06T03:54:32.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:32.191 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:32.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:32.381 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:32.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:32.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967672 -lt 1714967728 ]] 2024-05-06T03:54:32.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:32.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.575 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:32.575 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:32.577 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:32.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967672 -lt 1714967728 ]] 2024-05-06T03:54:32.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:32.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:32.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:32.769 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:32.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:32.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:32.772 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:32.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967672 -lt 1714967728 ]] 2024-05-06T03:54:32.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:32.773 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:32.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:32.963 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.963 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:32.964 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:32.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:32.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:32.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:32.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967672 -lt 1714967728 ]] 2024-05-06T03:54:32.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:32.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:36.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:36.164 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:36.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:36.165 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:36.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:36.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:36.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:36.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967676 -lt 1714967728 ]] 2024-05-06T03:54:36.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:36.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:36.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:36.360 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:36.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:36.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:36.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:36.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:36.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:36.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967676 -lt 1714967728 ]] 2024-05-06T03:54:36.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:36.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:36.555 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:36.555 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:36.556 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:36.556 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:36.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-05-06T03:54:36.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-05-06T03:54:36.558 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-05-06T03:54:36.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1714967676 -lt 1714967728 ]] 2024-05-06T03:54:36.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-05-06T03:54:36.560 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-05-06T03:54:43.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-05-06T03:54:43.529 INFO:tasks.workunit.client.0.smithi163.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:43.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7ebb8c79-f1ff-4dd2-8d36-b6a3743e908a","modified":"2024-05-06T03:53:44.351655Z","created":"2024-05-06T03:53:44.351655Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-05-06T03:54:43.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-05-06T03:54:43.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-05-06T03:54:43.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-05-06T03:54:43.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-05-06T03:54:43.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-05-06T03:54:43.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-05-06T03:54:43.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-05-06T03:54:43.888 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-05-06T03:54:43.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-05-06T03:54:44.166 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-05-06T03:54:44.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-05-06T03:54:44.408 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-05-06T03:54:44.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-05-06T03:54:47.556 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-05-06T03:54:47.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-05-06T03:54:47.821 INFO:tasks.workunit.client.0.smithi163.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-05-06T03:54:47.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-05-06T03:54:50.976 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-05-06T03:54:50.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-05-06T03:54:51.268 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.a 2024-05-06T03:54:51.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-05-06T03:54:51.473 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: removing mon.b would break quorum 2024-05-06T03:54:51.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-05-06T03:54:51.707 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: removing mon.c would break quorum 2024-05-06T03:54:51.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-05-06T03:54:57.857 INFO:tasks.workunit.client.0.smithi163.stderr:Error EBUSY: removing mon.d would break quorum 2024-05-06T03:54:57.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-05-06T03:55:01.053 INFO:tasks.workunit.client.0.smithi163.stderr:safe to remove mon.e 2024-05-06T03:55:01.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-05-06T03:55:01.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:55:01.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:55:01.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:55:01.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:55:01.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:55:01.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:55:01.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:55:01.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:55:01.177 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:55:01.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:55:01.178 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:55:01.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:55:01.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:55:01.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:55:01.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:55:01.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:55:01.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:55:01.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:55:01.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:55:01.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:55:01.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:55:01.192 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:55:01.192 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:01.192 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:55:01.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:55:01.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:55:01.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:55:01.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ok-to-stop 0 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:55:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:55:01.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:55:01.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:55:01.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:55:01.196 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:55:01.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:55:01.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:55:01.197 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:55:01.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:55:01.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:55:01.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:55:01.199 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:55:01.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:55:01.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-06T03:55:01.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-05-06T03:55:01.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:55:01.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:01.201 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52628 2024-05-06T03:55:01.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52628 2024-05-06T03:55:01.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:55:01.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:55:01.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-05-06T03:55:01.203 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-06T03:55:01.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T03:55:01.227 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-05-06T03:55:01.228 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-05-06T03:55:01.288 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T03:55:01.295 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/rados-striper 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-05-06T03:55:01.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none ' 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-05-06T03:55:01.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:55:01.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:55:01.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:55:01.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:55:01.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:55:01.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:55:01.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:55:01.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:55:01.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:55:01.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:55:01.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-05-06T03:55:01.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:55:01.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:01.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:01.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69482 2024-05-06T03:55:01.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:55:01.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:55:01.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-05-06T03:55:01.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:55:01.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:01.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:01.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69482 2024-05-06T03:55:01.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:55:01.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T03:55:01.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-05-06T03:55:01.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-05-06T03:55:01.783 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:55:01.783 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:55:01.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:55:01.784 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:55:01.784 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:01.784 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:01.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:55:01.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:55:01.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:55:01.978 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:55:01.979 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:01.979 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:01.979 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69482/ceph-mon.a.asok 2024-05-06T03:55:01.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:55:01.979 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69482/ceph-mon.a.asok config get fsid 2024-05-06T03:55:02.063 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69482/ceph-mon.a.asok 2024-05-06T03:55:02.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:55:02.065 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69482/ceph-mon.a.asok config get mon_host 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-05-06T03:55:02.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:55:02.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c08c17c-a241-4fd8-b9a5-dde5d95cf207 2024-05-06T03:55:02.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c08c17c-a241-4fd8-b9a5-dde5d95cf207' 2024-05-06T03:55:02.150 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 9c08c17c-a241-4fd8-b9a5-dde5d95cf207 2024-05-06T03:55:02.150 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:55:02.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWVDhmv01qCRAAp4XOAvqe2ZGPUvkrPxRRVg== 2024-05-06T03:55:02.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWVDhmv01qCRAAp4XOAvqe2ZGPUvkrPxRRVg=="}' 2024-05-06T03:55:02.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c08c17c-a241-4fd8-b9a5-dde5d95cf207 -i td/rados-striper/0/new.json 2024-05-06T03:55:03.015 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:55:03.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-05-06T03:55:03.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWVDhmv01qCRAAp4XOAvqe2ZGPUvkrPxRRVg== --osd-uuid 9c08c17c-a241-4fd8-b9a5-dde5d95cf207 2024-05-06T03:55:03.052 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.047+0000 7f0b32111640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:03.084 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.080+0000 7f0b32111640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:03.085 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.081+0000 7f0b32111640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:03.085 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.081+0000 7f0b32111640 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-05-06T03:55:03.086 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.081+0000 7f0b32111640 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-05-06T03:55:03.086 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.081+0000 7f0b32111640 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-05-06T03:55:03.116 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:03.112+0000 7f0b32111640 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-05-06T03:55:08.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-05-06T03:55:08.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:55:08.503 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:55:08.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:55:08.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:55:08.707 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:55:08.707 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:55:08.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:55:08.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:55:08.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:55:08.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:55:08.732 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:08.727+0000 7f7e2960e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:08.733 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:08.729+0000 7f7e2960e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:08.734 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:08.730+0000 7f7e2960e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:08.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:55:08.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:55:08.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:55:08.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:55:08.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:55:08.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:08.912 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:55:08.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:55:08.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:08.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:09.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:09.263 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:09.258+0000 7f7e2960e640 -1 Falling back to public interface 2024-05-06T03:55:10.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:10.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:10.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:55:10.122 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:55:10.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:10.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:10.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:11.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:11.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:11.318 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:55:11.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:55:11.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:11.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:11.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:11.633 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:11.628+0000 7f7e2960e640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:55:12.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:12.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:12.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:55:12.514 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:55:12.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:12.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:12.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:13.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:13.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:13.811 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:55:13.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:55:13.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:13.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:14.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:15.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:15.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:15.006 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:55:15.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:55:15.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:15.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:15.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:16.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:16.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:16.202 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:55:16.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:55:16.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:16.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:16.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:17.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:17.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:17.400 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:55:17.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:55:17.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:17.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:17.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:18.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:18.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:18.596 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:55:18.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:55:18.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:18.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:18.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:19.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:19.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:19.794 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:55:19.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:55:19.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:19.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:19.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:20.989 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:20.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:20.990 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:55:20.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:55:20.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:20.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:21.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:22.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:22.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:22.185 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:55:22.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:55:22.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:22.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:22.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:23.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:23.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:23.382 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:55:23.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:55:23.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:23.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:23.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:24.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:24.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:24.579 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:55:24.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:55:24.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:24.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:24.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:25.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:25.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:25.772 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:55:25.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:55:25.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:25.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:25.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:26.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:26.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:26.968 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:55:26.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:55:26.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:26.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:27.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:28.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:28.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:28.166 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:55:28.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:55:28.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:28.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:28.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:29.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:29.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:29.361 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:55:29.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:55:29.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:29.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:29.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:30.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:30.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:30.554 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T03:55:30.555 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T03:55:30.555 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:30.555 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:30.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:31.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:31.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:31.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T03:55:31.754 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T03:55:31.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:31.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:31.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:32.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:32.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:32.948 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T03:55:32.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T03:55:32.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:32.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:55:33.140 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3826852701,v1:127.0.0.1:6801/3826852701] [v2:127.0.0.1:6802/3826852701,v1:127.0.0.1:6803/3826852701] exists,up 9c08c17c-a241-4fd8-b9a5-dde5d95cf207 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T03:55:33.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:55:33.145 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:55:33.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84c0166a-5157-4619-98fd-cb325ebd0a6e 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 84c0166a-5157-4619-98fd-cb325ebd0a6e' 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 84c0166a-5157-4619-98fd-cb325ebd0a6e 2024-05-06T03:55:33.147 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:55:33.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1VDhmWB5ICRAAiXcyCCu+BvzLP71fJ4D5WQ== 2024-05-06T03:55:33.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1VDhmWB5ICRAAiXcyCCu+BvzLP71fJ4D5WQ=="}' 2024-05-06T03:55:33.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84c0166a-5157-4619-98fd-cb325ebd0a6e -i td/rados-striper/1/new.json 2024-05-06T03:55:33.445 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:55:33.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-05-06T03:55:33.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1VDhmWB5ICRAAiXcyCCu+BvzLP71fJ4D5WQ== --osd-uuid 84c0166a-5157-4619-98fd-cb325ebd0a6e 2024-05-06T03:55:33.481 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.477+0000 7fed0b6ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:33.482 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.478+0000 7fed0b6ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:33.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.479+0000 7fed0b6ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.480+0000 7fed0b6ac640 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-05-06T03:55:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.480+0000 7fed0b6ac640 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-05-06T03:55:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.480+0000 7fed0b6ac640 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-05-06T03:55:33.545 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:33.541+0000 7fed0b6ac640 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-05-06T03:55:39.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-05-06T03:55:39.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:55:39.023 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T03:55:39.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T03:55:39.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:55:39.228 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T03:55:39.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T03:55:39.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:55:39.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:55:39.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:55:39.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:55:39.252 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:39.248+0000 7f2e15811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:39.255 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:39.251+0000 7f2e15811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:39.256 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:39.252+0000 7f2e15811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:55:39.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T03:55:39.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:55:39.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T03:55:39.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:55:39.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:55:39.433 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:55:39.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:39.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:55:39.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:39.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:39.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:39.785 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:39.781+0000 7f2e15811640 -1 Falling back to public interface 2024-05-06T03:55:40.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:40.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:40.652 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:55:40.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:55:40.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:40.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:40.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:41.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:41.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:41.841 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:55:41.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:55:41.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:41.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:42.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:42.205 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:55:42.201+0000 7f2e15811640 -1 osd.1 0 log_to_monitors true 2024-05-06T03:55:43.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:43.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:43.028 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:55:43.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:55:43.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:43.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:43.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:44.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:44.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:44.323 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:55:44.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:55:44.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:44.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:44.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:45.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:45.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:45.518 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:55:45.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:55:45.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:45.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:45.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:46.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:46.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:46.716 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:55:46.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:55:46.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:46.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:46.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:47.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:47.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:47.912 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:55:47.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:55:47.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:47.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:48.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:49.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:49.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:49.108 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:55:49.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:55:49.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:49.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:49.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:50.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:50.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:50.303 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:55:50.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:55:50.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:50.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:50.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:51.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:51.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:51.501 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:55:51.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:55:51.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:51.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:51.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:52.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:52.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:52.697 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:55:52.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:55:52.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:52.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:52.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:53.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:53.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:53.898 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:55:53.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:55:53.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:53.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:54.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:55.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:55.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:55.092 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:55:55.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:55:55.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:55.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:55.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:56.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:56.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:56.289 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:55:56.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:55:56.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:56.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:56.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:57.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:57.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:57.487 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:55:57.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:55:57.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:57.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:57.678 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:58.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:58.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:58.680 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:55:58.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:55:58.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:58.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:55:58.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:55:59.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:55:59.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:55:59.875 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:55:59.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:55:59.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:55:59.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:56:00.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:01.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:01.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:01.069 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T03:56:01.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T03:56:01.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:01.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:56:01.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:02.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:02.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:02.259 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T03:56:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T03:56:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:56:02.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:03.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:03.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:03.495 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T03:56:03.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T03:56:03.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:03.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T03:56:03.690 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1712185859,v1:127.0.0.1:6809/1712185859] [v2:127.0.0.1:6810/1712185859,v1:127.0.0.1:6811/1712185859] exists,up 84c0166a-5157-4619-98fd-cb325ebd0a6e 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T03:56:03.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:56:03.695 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:56:03.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=061cac14-a443-4645-93fb-0865993d6547 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 061cac14-a443-4645-93fb-0865993d6547' 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 061cac14-a443-4645-93fb-0865993d6547 2024-05-06T03:56:03.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:56:03.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTVDhm6acWKhAAOKqxOwNkTQzo+3sgigG3sQ== 2024-05-06T03:56:03.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTVDhm6acWKhAAOKqxOwNkTQzo+3sgigG3sQ=="}' 2024-05-06T03:56:03.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 061cac14-a443-4645-93fb-0865993d6547 -i td/rados-striper/2/new.json 2024-05-06T03:56:03.996 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:56:04.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-05-06T03:56:04.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTVDhm6acWKhAAOKqxOwNkTQzo+3sgigG3sQ== --osd-uuid 061cac14-a443-4645-93fb-0865993d6547 2024-05-06T03:56:04.032 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.028+0000 7f925ec16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:04.033 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.030+0000 7f925ec16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:04.034 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.030+0000 7f925ec16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:04.035 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.031+0000 7f925ec16640 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-05-06T03:56:04.035 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.031+0000 7f925ec16640 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-05-06T03:56:04.035 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.031+0000 7f925ec16640 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-05-06T03:56:04.105 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:04.101+0000 7f925ec16640 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-05-06T03:56:09.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-05-06T03:56:09.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:56:09.824 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T03:56:09.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T03:56:09.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:56:10.030 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T03:56:10.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T03:56:10.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=de1dd144-639e-4ec2-9f8d-16425db9eaa2 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.69482/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:56:10.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:56:10.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:56:10.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:56:10.054 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:10.050+0000 7f833591e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:10.056 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:10.052+0000 7f833591e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:10.057 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:10.054+0000 7f833591e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:10.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:56:10.234 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:56:10.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:10.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:10.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:10.586 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:10.582+0000 7f833591e640 -1 Falling back to public interface 2024-05-06T03:56:11.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:11.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:11.444 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:56:11.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:56:11.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:11.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:11.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:12.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:12.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:12.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:56:12.641 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:56:12.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:12.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:12.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:13.014 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:13.010+0000 7f833591e640 -1 osd.2 0 log_to_monitors true 2024-05-06T03:56:13.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:13.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:13.830 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:56:13.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:56:13.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:13.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:14.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:15.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:15.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:15.114 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:56:15.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:56:15.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:15.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:15.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:16.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:16.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:16.309 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:56:16.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:56:16.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:16.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:16.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:17.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:17.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:17.507 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:56:17.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:56:17.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:17.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:17.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:18.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:18.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:18.702 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:56:18.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:56:18.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:18.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:18.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:19.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:19.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:19.899 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:56:19.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:56:19.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:19.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:20.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:21.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:21.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:21.096 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:56:21.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:56:21.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:21.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:21.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:22.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:22.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:22.293 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:56:22.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:56:22.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:22.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:22.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:23.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:23.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:23.490 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:56:23.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:56:23.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:23.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:23.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:24.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:24.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:24.689 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:56:24.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:56:24.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:24.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:24.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:25.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:25.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:25.879 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:56:25.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:56:25.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:25.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:26.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:27.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:27.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:27.072 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:56:27.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:56:27.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:27.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:27.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:28.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:28.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:28.263 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:56:28.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:56:28.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:28.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:28.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:29.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:29.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:29.453 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:56:29.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:56:29.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:29.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:29.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:30.651 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:56:30.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:56:30.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:30.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:30.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:31.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:31.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:31.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T03:56:31.846 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T03:56:31.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:31.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:32.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:33.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:33.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:33.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T03:56:33.042 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T03:56:33.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:33.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:33.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:34.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:34.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:34.236 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T03:56:34.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T03:56:34.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:34.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T03:56:34.430 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1204479124,v1:127.0.0.1:6817/1204479124] [v2:127.0.0.1:6818/1204479124,v1:127.0.0.1:6819/1204479124] exists,up 061cac14-a443-4645-93fb-0865993d6547 2024-05-06T03:56:34.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:56:34.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:56:34.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:56:34.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-05-06T03:56:34.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T03:56:34.618 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T03:56:34.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T03:56:34.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T03:56:34.926 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T03:56:34.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T03:56:35.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T03:56:37.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-05-06T03:56:37.447 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records in 2024-05-06T03:56:37.447 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records out 2024-05-06T03:56:37.447 INFO:tasks.workunit.client.0.smithi163.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.5741e-05 s, 22.1 MB/s 2024-05-06T03:56:37.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-05-06T03:56:38.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-05-06T03:56:38.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-05-06T03:56:38.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-05-06T03:56:38.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-05-06T03:56:38.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-05-06T03:56:38.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-05-06T03:56:38.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-05-06T03:56:38.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-05-06T03:56:38.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-05-06T03:56:38.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-05-06T03:56:38.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-05-06T03:56:38.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-05-06T03:56:38.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-05-06T03:56:38.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-05-06T03:56:39.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-05-06T03:56:39.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-05-06T03:56:39.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-05-06T03:56:39.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-05-06T03:56:39.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-05-06T03:56:39.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-05-06T03:56:39.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-05-06T03:56:39.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-05-06T03:56:39.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-05-06T03:56:39.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-05-06T03:56:39.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-05-06T03:56:39.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-05-06T03:56:39.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-05-06T03:56:39.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-05-06T03:56:39.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-05-06T03:56:39.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-05-06T03:56:39.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-05-06T03:56:39.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-05-06T03:56:39.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-05-06T03:56:39.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-05-06T03:56:39.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-05-06T03:56:39.485 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-05-06T03:56:39.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-05-06T03:56:39.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-05-06T03:56:39.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-05-06T03:56:39.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:56:39.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No data available' 2024-05-06T03:56:39.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:56:39.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:56:39.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-05-06T03:56:39.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-05-06T03:56:39.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-05-06T03:56:39.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-05-06T03:56:39.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-05-06T03:56:39.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-05-06T03:56:39.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-05-06T03:56:39.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:56:39.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No data available' 2024-05-06T03:56:39.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:56:39.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:56:39.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-05-06T03:56:39.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-05-06T03:56:39.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-05-06T03:56:39.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-05-06T03:56:39.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-05-06T03:56:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-05-06T03:56:39.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-05-06T03:56:39.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-05-06T03:56:39.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:56:39.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No such file or directory' 2024-05-06T03:56:39.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:56:39.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:56:39.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd --striper stat toyfile 2024-05-06T03:56:39.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-05-06T03:56:39.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-05-06T03:56:39.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-05-06T03:56:39.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-05-06T03:56:39.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-05-06T03:56:39.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/rados-striper 2024-05-06T03:56:39.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:56:39.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local 'expected=No such file or directory' 2024-05-06T03:56:39.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:56:39.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:56:39.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-05-06T03:56:39.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-05-06T03:56:39.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-05-06T03:56:39.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-05-06T03:56:39.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-05-06T03:56:39.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-05-06T03:56:39.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:56:39.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:56:39.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:56:39.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:56:39.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:56:39.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:56:39.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:56:39.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:56:39.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:56:39.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:56:39.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:56:39.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:56:39.926 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:56:39.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:56:39.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:56:39.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-05-06T03:56:39.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:56:39.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:39.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:56:39.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69482 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/rados-striper 0 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-06T03:56:39.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:56:39.941 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:56:39.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:56:39.942 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:56:39.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:56:39.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:56:39.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:56:39.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:56:39.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:56:39.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:56:39.945 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:56:39.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:56:39.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-06T03:56:39.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-05-06T03:56:39.947 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:56:39.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:39.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69482 2024-05-06T03:56:39.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69482 2024-05-06T03:56:39.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:56:39.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:56:39.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-05-06T03:56:39.950 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-06T03:56:39.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T03:56:40.013 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-05-06T03:56:40.014 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-05-06T03:56:40.076 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: run_tests: export .:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2344: export: `.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin': not a valid identifier 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: run_tests: export CEPH_MON=127.0.0.1:7109 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: run_tests: CEPH_MON=127.0.0.1:7109 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: run_tests: export CEPH_ARGS 2024-05-06T03:56:40.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: uuidgen 2024-05-06T03:56:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: run_tests: CEPH_ARGS+=' --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none ' 2024-05-06T03:56:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-05-06T03:56:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: export CEPH_CONF=/dev/null 2024-05-06T03:56:40.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: run_tests: CEPH_CONF=/dev/null 2024-05-06T03:56:40.081 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: run_tests: set 2024-05-06T03:56:40.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: run_tests: local 'funcs=test_activate_osd 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_activate_osd_after_mark_down 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_activate_osd_skip_benchmark 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_destroy_osd 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_display_logs 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_erasure_code_plugin_exists 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_expect_failure 2024-05-06T03:56:40.084 INFO:tasks.workunit.client.0.smithi163.stderr:test_flush_pg_stats 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_config 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_is_making_recovery_progress 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_last_scrub_stamp 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_not_primary 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_num_active_clean 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_num_active_or_peered 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_num_pgs 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_op_scheduler 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_osd_id_used_by_pgs 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_osds 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_pg 2024-05-06T03:56:40.085 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_primary 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_get_timeout_delays 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_is_clean 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_kill_daemon 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_kill_daemons 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_objectstore_tool 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_pg_schedule_scrub 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_pg_scrub 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_repair 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_run_in_background 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_run_mon 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_run_osd 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_set_config 2024-05-06T03:56:40.086 INFO:tasks.workunit.client.0.smithi163.stderr:test_setup 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_teardown 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_background 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_for_clean 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_for_health_ok 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_for_osd 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_for_peered 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_for_scrub 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:test_wait_osd_id_used_by_pgs' 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: run_tests: local dir=td/ceph-helpers 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_activate_osd td/ceph-helpers 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-05-06T03:56:40.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:56:40.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:56:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:56:40.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:56:40.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:40.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:40.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:56:40.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:56:40.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:56:40.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:56:40.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:56:40.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:40.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:40.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:56:40.094 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:56:40.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T03:56:40.567 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:56:40.567 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:56:40.567 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:56:40.568 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:56:40.568 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:40.568 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:40.568 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:56:40.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:56:40.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:56:40.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:56:40.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:56:40.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:56:40.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:56:40.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:56:40.842 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:56:40.843 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:56:40.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-05-06T03:56:40.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:56:40.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:56:40.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:56:40.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:56:40.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:56:40.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:56:41.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:56:41.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:56:41.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:56:41.805 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:56:41.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:56:41.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:56:41.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:56:41.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b0760d2-6bbf-4d26-beee-c44b4c0b62c1 2024-05-06T03:56:41.807 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 8b0760d2-6bbf-4d26-beee-c44b4c0b62c1 2024-05-06T03:56:41.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8b0760d2-6bbf-4d26-beee-c44b4c0b62c1' 2024-05-06T03:56:41.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:56:41.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5VDhmXLTSMBAAYK2Sxpc0A+KhjFtSqXt9ug== 2024-05-06T03:56:41.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5VDhmXLTSMBAAYK2Sxpc0A+KhjFtSqXt9ug=="}' 2024-05-06T03:56:41.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b0760d2-6bbf-4d26-beee-c44b4c0b62c1 -i td/ceph-helpers/0/new.json 2024-05-06T03:56:42.111 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:56:42.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T03:56:42.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5VDhmXLTSMBAAYK2Sxpc0A+KhjFtSqXt9ug== --osd-uuid 8b0760d2-6bbf-4d26-beee-c44b4c0b62c1 2024-05-06T03:56:42.150 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.146+0000 7f3350417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:42.178 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.174+0000 7f3350417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:42.178 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.175+0000 7f3350417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:42.179 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.175+0000 7f3350417640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:56:42.179 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.175+0000 7f3350417640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:56:42.179 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.175+0000 7f3350417640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:56:42.218 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:42.214+0000 7f3350417640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T03:56:47.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T03:56:47.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:56:47.930 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:56:47.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:56:47.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:56:48.495 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:56:48.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:56:48.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:56:48.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:56:48.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:56:48.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:56:48.521 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:48.517+0000 7faafc81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:48.526 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:48.521+0000 7faafc81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:48.527 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:48.523+0000 7faafc81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:56:48.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:56:48.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:56:48.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:56:48.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:56:48.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:56:48.940 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:56:48.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:48.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:56:48.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:48.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:49.054 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:49.051+0000 7faafc81d640 -1 Falling back to public interface 2024-05-06T03:56:49.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:50.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:50.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:50.367 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:56:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:56:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:50.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:51.411 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:56:51.407+0000 7faafc81d640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:56:51.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:51.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:51.802 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:56:51.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:56:51.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:51.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:52.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:53.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:53.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:53.207 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:56:53.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:56:53.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:53.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:53.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:54.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:54.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:54.636 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:56:54.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:56:54.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:54.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:55.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:56.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:56.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:56.210 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:56:56.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:56:56.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:56.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:56.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:57.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:57.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:57.630 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:56:57.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:56:57.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:57.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:58.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:56:59.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:56:59.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:56:59.217 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:56:59.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:56:59.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:56:59.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:56:59.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:00.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:00.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:00.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:57:00.650 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:57:00.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:00.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:01.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:02.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:02.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:02.208 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:57:02.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:57:02.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:02.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:02.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:03.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:03.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:03.636 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:57:03.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:57:03.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:03.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:04.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:05.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:05.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:05.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:57:05.143 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:57:05.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:05.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:05.707 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:06.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:06.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:06.709 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:57:06.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:57:06.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:06.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:07.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:08.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:08.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:08.420 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:57:08.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:57:08.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:08.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:09.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:10.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:10.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:10.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:57:10.466 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:57:10.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:10.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:11.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:12.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:12.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:12.269 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:57:12.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:57:12.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:12.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:13.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:14.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:14.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:14.157 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:57:14.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:57:14.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:14.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:14.576 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2996869909,v1:127.0.0.1:6803/2996869909] [v2:127.0.0.1:6804/2996869909,v1:127.0.0.1:6805/2996869909] exists,up 8b0760d2-6bbf-4d26-beee-c44b4c0b62c1 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:14.578 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:14.579 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:57:14.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-05-06T03:57:14.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:57:14.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-05-06T03:57:14.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-05-06T03:57:14.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-05-06T03:57:14.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-05-06T03:57:14.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:57:14.664 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:57:14.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:57:14.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:57:14.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:57:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:57:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-05-06T03:57:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T03:57:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T03:57:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T03:57:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:57:14.973 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T03:57:14.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-05-06T03:57:14.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:57:14.975 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T03:57:14.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T03:57:14.998 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:14.994+0000 7f313f60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:15.000 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:14.996+0000 7f313f60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:15.001 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:14.997+0000 7f313f60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:15.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:57:15.424 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:57:15.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:15.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:15.525 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:15.522+0000 7f313f60f640 -1 Falling back to public interface 2024-05-06T03:57:15.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:16.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:16.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:16.846 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:57:16.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:57:16.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:16.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:17.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:18.289 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:18.284+0000 7f313f60f640 -1 osd.0 5 log_to_monitors true 2024-05-06T03:57:18.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:18.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:18.307 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:57:18.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:57:18.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:18.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:18.736 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1702390762,v1:127.0.0.1:6803/1702390762] [v2:127.0.0.1:6804/1702390762,v1:127.0.0.1:6805/1702390762] exists,up 8b0760d2-6bbf-4d26-beee-c44b4c0b62c1 2024-05-06T03:57:18.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:57:18.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:57:18.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:57:18.738 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-05-06T03:57:18.738 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:57:18.738 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:57:18.739 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:18.739 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:18.739 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:18.739 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:57:18.739 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-05-06T03:57:18.739 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:57:18.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:57:18.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:57:18.929 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:57:18.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:57:18.930 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:57:18.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:57:18.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:57:18.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:57:18.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:18.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:57:18.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:57:18.932 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:18.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:57:18.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:57:18.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:57:18.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:57:18.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:18.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:18.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:57:18.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:57:18.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:57:18.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:57:18.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:57:18.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:57:18.943 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:57:18.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:57:18.943 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:57:18.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:57:18.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:57:18.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:57:18.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:18.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:57:18.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:57:18.945 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:18.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:57:18.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:57:18.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:57:18.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:57:18.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:18.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:18.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:57:18.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:57:18.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:57:18.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:57:18.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:57:18.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:18.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:18.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:57:18.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T03:57:19.406 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:57:19.406 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:57:19.406 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:57:19.406 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:57:19.406 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:19.407 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:19.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:19.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:57:19.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:19.601 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:19.602 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:57:19.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:57:19.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:57:19.682 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:57:19.682 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:57:19.682 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:57:19.682 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:57:19.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:57:19.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:57:20.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:57:20.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:57:20.606 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:57:20.606 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:57:20.606 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:20.606 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:20.606 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:20.606 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:57:20.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:57:20.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-05-06T03:57:20.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T03:57:20.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:57:20.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:57:20.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:57:20.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:57:20.637 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:57:20.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56356598-121e-4266-9ce4-92abee937b66 2024-05-06T03:57:20.639 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 56356598-121e-4266-9ce4-92abee937b66 2024-05-06T03:57:20.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 56356598-121e-4266-9ce4-92abee937b66' 2024-05-06T03:57:20.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:57:20.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgVThmS9HKJhAA0E4bezTHZ68mp9c3+6YZ8w== 2024-05-06T03:57:20.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgVThmS9HKJhAA0E4bezTHZ68mp9c3+6YZ8w=="}' 2024-05-06T03:57:20.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56356598-121e-4266-9ce4-92abee937b66 -i td/ceph-helpers/0/new.json 2024-05-06T03:57:20.926 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:57:20.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T03:57:20.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgVThmS9HKJhAA0E4bezTHZ68mp9c3+6YZ8w== --osd-uuid 56356598-121e-4266-9ce4-92abee937b66 2024-05-06T03:57:20.962 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:20.958+0000 7fbbb9410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:20.992 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:20.988+0000 7fbbb9410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:20.993 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:20.989+0000 7fbbb9410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:20.993 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:20.990+0000 7fbbb9410640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:57:20.993 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:20.990+0000 7fbbb9410640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:57:20.993 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:20.990+0000 7fbbb9410640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:57:21.033 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:21.028+0000 7fbbb9410640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T03:57:26.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T03:57:26.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:57:26.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:57:26.521 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:57:26.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:57:27.133 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:57:27.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:57:27.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:57:27.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:57:27.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:57:27.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:57:27.160 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:27.157+0000 7f4abecfb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:27.162 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:27.159+0000 7f4abecfb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:27.163 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:27.160+0000 7f4abecfb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:27.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:57:27.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:57:27.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:57:27.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:57:27.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:57:27.560 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:57:27.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:27.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:57:27.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:27.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:27.677 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:27.674+0000 7f4abecfb640 -1 Falling back to public interface 2024-05-06T03:57:27.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:28.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:28.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:28.996 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:57:28.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:57:28.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:28.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:29.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:30.026 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:30.022+0000 7f4abecfb640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:57:30.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:30.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:30.422 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:57:30.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:57:30.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:30.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:30.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:31.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:31.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:31.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:57:31.828 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:57:31.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:31.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:32.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:33.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:33.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:33.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:57:33.258 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:57:33.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:33.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:33.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:34.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:34.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:34.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:57:34.825 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:57:34.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:34.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:35.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:36.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:36.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:36.239 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:57:36.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:57:36.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:36.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:36.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:37.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:37.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:37.649 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:57:37.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:57:37.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:37.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:38.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:39.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:39.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:39.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:57:39.119 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:57:39.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:39.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:39.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:40.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:40.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:40.538 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:57:40.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:57:40.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:40.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:40.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:41.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:41.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:41.961 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:57:41.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:57:41.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:41.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:42.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:43.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:43.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:43.690 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:57:43.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:57:43.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:43.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:44.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:45.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:45.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:45.260 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:57:45.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:57:45.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:45.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:46.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:47.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:47.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:47.003 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:57:47.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:57:47.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:47.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:48.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:49.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:49.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:49.225 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:57:49.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:57:49.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:49.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:50.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:51.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:51.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:51.027 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:57:51.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:57:51.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:51.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:52.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:53.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:53.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:53.063 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:57:53.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:57:53.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:53.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:53.499 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1496359231,v1:127.0.0.1:6803/1496359231] [v2:127.0.0.1:6804/1496359231,v1:127.0.0.1:6805/1496359231] exists,up 56356598-121e-4266-9ce4-92abee937b66 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:57:53.501 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:53.502 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:53.502 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:53.502 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:57:53.502 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-05-06T03:57:53.502 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:57:53.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-05-06T03:57:53.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-05-06T03:57:53.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-05-06T03:57:53.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-05-06T03:57:53.587 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:57:53.587 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:57:53.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:57:53.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:57:53.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:57:53.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:57:53.893 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-05-06T03:57:54.315 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already down. 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:54.327 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:57:54.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:57:54.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:54.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-05-06T03:57:54.753 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1496359231,v1:127.0.0.1:6803/1496359231] [v2:127.0.0.1:6804/1496359231,v1:127.0.0.1:6805/1496359231] exists 56356598-121e-4266-9ce4-92abee937b66 2024-05-06T03:57:54.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:57:54.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:57:54.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:57:54.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T03:57:54.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:54.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:54.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:57:54.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:57:54.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:57:54.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:57:54.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:57:54.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:57:54.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:57:54.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T03:57:54.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-05-06T03:57:54.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:57:54.762 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:57:54.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T03:57:54.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-05-06T03:57:54.764 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-05-06T03:57:54.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T03:57:54.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T03:57:54.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T03:57:54.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T03:57:54.782 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:54.778+0000 7f818a82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:54.784 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:54.780+0000 7f818a82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:54.785 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:54.781+0000 7f818a82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:57:55.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T03:57:55.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:57:55.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:57:55.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:57:55.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:57:55.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:55.181 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:57:55.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:57:55.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:55.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:55.300 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:55.296+0000 7f818a82c640 -1 Falling back to public interface 2024-05-06T03:57:55.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:56.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:56.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:56.607 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:57:56.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:57:56.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:56.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:57:58.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:57:58.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:57:58.034 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:57:58.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:57:58.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:57:58.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:57:58.098 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:57:58.095+0000 7f818a82c640 -1 osd.0 5 log_to_monitors true 2024-05-06T03:57:58.504 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/3829502275,v1:127.0.0.1:6803/3829502275] [v2:127.0.0.1:6804/3829502275,v1:127.0.0.1:6805/3829502275] exists,up 56356598-121e-4266-9ce4-92abee937b66 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:58.506 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:58.507 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:57:58.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-05-06T03:57:58.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:57:58.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:57:58.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:57:58.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:57:58.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:57:58.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:57:58.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:57:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:57:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:57:58.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:57:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:57:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:57:58.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:58.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:57:58.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:57:58.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:57:58.707 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:57:58.708 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:58.708 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:58.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:57:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:57:58.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:57:58.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:57:58.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:57:58.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:57:58.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:57:58.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:57:58.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:57:58.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:57:58.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:57:58.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:57:58.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:57:58.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:57:58.713 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:57:58.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:58.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:57:58.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:57:58.714 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:57:58.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:57:58.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:57:58.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:57:58.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:57:58.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:58.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:58.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:57:58.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:57:58.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:57:58.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:57:58.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:57:58.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:58.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:58.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:57:58.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:57:58.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:57:59.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:57:59.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:57:59.351 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:57:59.351 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:57:59.351 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:59.352 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:57:59.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:57:59.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:57:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:57:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:57:59.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:57:59.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:58:00.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:58:00.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:00.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:00.364 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:00.364 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:00.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:00.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:00.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:58:00.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:58:00.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-05-06T03:58:00.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T03:58:00.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:58:00.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:58:00.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:58:00.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:58:00.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:58:00.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 2024-05-06T03:58:00.399 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 2024-05-06T03:58:00.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2' 2024-05-06T03:58:00.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:58:00.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIVThmNNlyGBAAnYK8EFW8+7qvJQ9VTLYDzw== 2024-05-06T03:58:00.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIVThmNNlyGBAAnYK8EFW8+7qvJQ9VTLYDzw=="}' 2024-05-06T03:58:00.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 -i td/ceph-helpers/0/new.json 2024-05-06T03:58:00.691 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:58:00.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T03:58:00.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQBIVThmNNlyGBAAnYK8EFW8+7qvJQ9VTLYDzw== --osd-uuid f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 2024-05-06T03:58:00.732 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.727+0000 7f11ec60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:00.760 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.755+0000 7f11ec60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:00.761 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.757+0000 7f11ec60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:00.761 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.757+0000 7f11ec60f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:58:00.762 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.757+0000 7f11ec60f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:58:00.762 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.757+0000 7f11ec60f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:58:00.817 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:00.812+0000 7f11ec60f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T03:58:06.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T03:58:06.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:58:06.292 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:58:06.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:58:06.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:58:06.893 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:58:06.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:58:06.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-05-06T03:58:06.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:58:06.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:58:06.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:58:06.921 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:06.917+0000 7f8472615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:06.923 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:06.920+0000 7f8472615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:06.924 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:06.921+0000 7f8472615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:07.321 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:58:07.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:58:07.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:07.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:07.441 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:07.437+0000 7f8472615640 -1 Falling back to public interface 2024-05-06T03:58:07.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:08.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:08.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:08.756 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:58:08.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:58:08.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:08.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:09.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:09.799 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:09.796+0000 7f8472615640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:58:10.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:10.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:10.199 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:58:10.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:58:10.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:10.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:10.638 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3453344178,v1:127.0.0.1:6803/3453344178] [v2:127.0.0.1:6804/3453344178,v1:127.0.0.1:6805/3453344178] exists,up f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 2024-05-06T03:58:10.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:10.642 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:58:10.642 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-05-06T03:58:10.642 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-05-06T03:58:10.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-05-06T03:58:10.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-05-06T03:58:10.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:58:10.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:58:10.728 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:10.728 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:10.729 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:10.729 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:58:10.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-05-06T03:58:10.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:10.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:12.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:12.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-05-06T03:58:12.587 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already down. 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:58:12.599 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:58:12.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:12.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-05-06T03:58:13.059 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/3453344178,v1:127.0.0.1:6803/3453344178] [v2:127.0.0.1:6804/3453344178,v1:127.0.0.1:6805/3453344178] exists f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 2024-05-06T03:58:13.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:58:13.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:58:13.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:58:13.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-05-06T03:58:13.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T03:58:13.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:58:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:58:13.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T03:58:13.068 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:58:13.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-05-06T03:58:13.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-05-06T03:58:13.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T03:58:13.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T03:58:13.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T03:58:13.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T03:58:13.089 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:13.086+0000 7f338fc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:13.091 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:13.087+0000 7f338fc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:13.092 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:13.089+0000 7f338fc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:13.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T03:58:13.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:58:13.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:13.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:13.614 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:13.610+0000 7f338fc1d640 -1 Falling back to public interface 2024-05-06T03:58:13.920 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:14.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:14.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:14.923 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:58:14.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:58:14.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:14.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:15.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:16.229 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:16.226+0000 7f338fc1d640 -1 osd.0 6 log_to_monitors true 2024-05-06T03:58:16.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:16.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:16.347 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:58:16.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:58:16.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:16.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:16.835 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/3671132411,v1:127.0.0.1:6803/3671132411] [v2:127.0.0.1:6804/3671132411,v1:127.0.0.1:6805/3671132411] exists,up f1b21f6f-cd6b-4f51-aa35-d68b5936a1c2 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:16.837 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:16.838 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:16.838 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:58:16.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-05-06T03:58:16.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-05-06T03:58:16.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-05-06T03:58:16.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-05-06T03:58:16.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-05-06T03:58:16.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-05-06T03:58:17.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:17.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:17.117 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:17.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:17.118 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:17.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:17.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:17.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:17.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:17.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:17.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:17.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:17.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:17.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:17.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:17.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:17.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:17.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:17.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_destroy_osd td/ceph-helpers 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:58:17.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:17.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:17.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:17.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:17.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:17.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:17.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:17.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:17.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:17.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:17.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:17.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:17.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:17.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:17.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:17.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:17.134 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:17.134 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:17.134 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:17.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:17.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:17.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:17.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:58:17.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:58:17.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:17.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:17.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:58:17.138 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:58:17.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:58:17.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T03:58:17.572 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:17.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:58:17.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:58:17.775 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:17.776 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:17.776 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:17.776 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:58:17.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:58:17.776 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:58:17.855 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:58:17.855 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:58:17.855 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:58:17.856 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:58:17.856 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:58:17.856 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:58:17.856 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:58:17.856 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:58:17.856 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:17.857 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:17.857 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:17.857 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:58:17.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:58:17.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:58:17.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-05-06T03:58:17.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:58:17.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:58:17.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:58:17.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:58:17.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:58:17.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:58:18.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:58:18.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:18.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:18.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:18.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:18.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:18.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:18.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:58:18.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:58:18.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-05-06T03:58:18.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:58:18.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:58:18.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:58:18.821 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:58:18.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c75c821-3b17-4e41-b71a-76b381fee394 2024-05-06T03:58:18.822 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 5c75c821-3b17-4e41-b71a-76b381fee394 2024-05-06T03:58:18.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5c75c821-3b17-4e41-b71a-76b381fee394' 2024-05-06T03:58:18.822 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:58:18.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaVThmtL+zMRAACiXmwZS5j1q1B+XoyXf9dw== 2024-05-06T03:58:18.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaVThmtL+zMRAACiXmwZS5j1q1B+XoyXf9dw=="}' 2024-05-06T03:58:18.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c75c821-3b17-4e41-b71a-76b381fee394 -i td/ceph-helpers/0/new.json 2024-05-06T03:58:19.115 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:58:19.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T03:58:19.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaVThmtL+zMRAACiXmwZS5j1q1B+XoyXf9dw== --osd-uuid 5c75c821-3b17-4e41-b71a-76b381fee394 2024-05-06T03:58:19.154 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.150+0000 7fb33280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:19.183 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.180+0000 7fb33280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:19.184 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.181+0000 7fb33280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:19.185 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.181+0000 7fb33280f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:58:19.185 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.181+0000 7fb33280f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:58:19.185 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.181+0000 7fb33280f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:58:19.223 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:19.220+0000 7fb33280f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T03:58:24.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T03:58:24.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:58:24.754 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:58:24.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:58:24.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:58:25.332 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:58:25.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:58:25.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:58:25.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:58:25.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:58:25.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:58:25.359 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:25.355+0000 7f29b2263640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:25.363 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:25.359+0000 7f29b2263640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:25.364 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:25.360+0000 7f29b2263640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:58:25.785 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:58:25.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:25.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:25.895 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:25.892+0000 7f29b2263640 -1 Falling back to public interface 2024-05-06T03:58:26.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:27.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:27.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:27.212 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:58:27.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:58:27.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:27.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:27.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:28.265 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:58:28.261+0000 7f29b2263640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:58:28.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:28.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:28.663 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:58:28.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:58:28.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:28.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:29.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:30.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:30.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:30.291 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:58:30.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:58:30.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:30.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:30.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:31.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:31.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:31.692 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:58:31.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:58:31.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:31.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:32.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:33.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:33.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:33.184 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:58:33.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:58:33.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:33.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:33.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:34.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:34.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:34.590 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:58:34.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:58:34.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:34.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:35.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:36.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:36.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:36.039 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:58:36.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:58:36.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:36.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:36.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:37.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:37.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:37.472 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:58:37.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:58:37.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:37.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:37.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:38.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:38.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:38.996 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:58:38.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:58:38.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:38.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:39.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:40.402 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:58:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:58:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:40.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:41.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:41.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:41.983 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:58:41.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:58:41.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:41.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:42.441 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:43.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:43.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:43.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:58:43.443 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:58:43.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:43.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:44.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:45.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:45.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:45.154 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:58:45.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:58:45.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:45.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:45.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:46.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:46.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:46.854 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:58:46.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:58:46.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:46.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:47.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:48.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:48.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:48.299 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:58:48.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:58:48.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:48.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:49.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:50.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:50.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:50.538 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:58:50.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:58:50.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:50.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:51.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:58:52.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:58:52.203 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:58:52.203 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T03:58:52.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T03:58:52.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:58:52.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:58:52.643 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3530890474,v1:127.0.0.1:6803/3530890474] [v2:127.0.0.1:6804/3530890474,v1:127.0.0.1:6805/3530890474] exists,up 5c75c821-3b17-4e41-b71a-76b381fee394 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-05-06T03:58:52.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-05-06T03:58:53.173 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already out. 2024-05-06T03:58:53.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-05-06T03:58:53.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:53.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:53.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:53.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:53.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:53.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:53.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-05-06T03:58:53.910 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already down. 2024-05-06T03:58:53.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-05-06T03:58:54.362 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 does not exist 2024-05-06T03:58:54.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-05-06T03:58:54.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-05-06T03:58:54.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:54.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-05-06T03:58:54.374 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:54.375 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:54.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:54.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:54.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:54.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:54.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:54.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:54.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:54.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:54.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:54.378 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:54.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:54.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:54.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:54.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:54.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:54.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:54.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-05-06T03:58:54.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:54.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:54.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:54.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:54.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:54.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:54.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-05-06T03:58:54.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-05-06T03:58:54.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-05-06T03:58:54.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-05-06T03:58:54.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:54.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:54.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:54.823 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:54.823 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:54.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:54.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:54.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:54.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:54.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:54.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:54.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:54.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:54.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:54.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:54.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:54.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:54.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:54.926 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:54.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:54.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:54.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:54.934 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:54.934 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:54.934 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:54.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:54.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:54.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_display_logs td/ceph-helpers 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: test_display_logs: local dir=td/ceph-helpers 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: test_display_logs: setup td/ceph-helpers 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:54.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:54.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:54.937 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:54.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:54.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:54.937 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:54.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:54.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:54.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:54.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:54.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:54.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:54.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:58:54.941 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:58:54.941 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:54.941 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:54.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: test_display_logs: run_mon td/ceph-helpers a 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:58:54.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:58:54.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:58:54.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:58:54.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T03:58:55.436 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:58:55.436 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:55.436 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:55.436 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:55.436 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:55.436 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:55.437 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:55.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:58:55.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:58:55.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:58:55.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:58:55.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:58:55.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:58:55.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:58:55.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:58:55.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:58:55.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:58:55.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:58:55.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:58:55.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:58:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:58:55.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: test_display_logs: kill_daemons td/ceph-helpers 2024-05-06T03:58:55.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:55.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:55.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:55.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:55.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:57.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:57.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: test_display_logs: display_logs td/ceph-helpers 2024-05-06T03:58:57.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: display_logs: local dir=td/ceph-helpers 2024-05-06T03:58:57.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-05-06T03:58:57.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: display_logs: read file 2024-05-06T03:58:57.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-05-06T03:58:57.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: display_logs: cat td/ceph-helpers/mon.a.log 2024-05-06T03:58:57.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: display_logs: read file 2024-05-06T03:58:57.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: test_display_logs: teardown td/ceph-helpers 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:57.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:57.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:57.104 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:57.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:57.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:57.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:57.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:57.107 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:57.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:57.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:57.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:57.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:57.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:57.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:57.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:57.112 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:57.112 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:57.112 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:57.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:57.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:57.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2091: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:57.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:57.116 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:57.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:57.117 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:57.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:57.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:57.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:57.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:57.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:57.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:57.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:57.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:57.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:57.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:57.123 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:57.123 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:57.123 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:57.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:57.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:57.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:57.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:58:57.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:58:57.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:57.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:57.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:58:57.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:58:57.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:58:57.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:57.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T03:58:57.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:58:57.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:58:57.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:58:57.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:58:57.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:58:57.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:58:57.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:58:57.850 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:58:57.851 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:57.851 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:57.851 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:58:57.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:58:57.851 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:58:57.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2093: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-05-06T03:58:57.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:58:57.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:58:57.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:58:57.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:58:57.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:58:57.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:58:58.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:58:58.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:58:58.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:58:58.791 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:58:58.792 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:58.792 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:58.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:58:58.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:58:58.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: erasure_code_plugin_exists: local plugin=jerasure 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: local status 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local grepstr 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: local s 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: case `uname` in 2024-05-06T03:58:58.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: uname 2024-05-06T03:58:58.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-05-06T03:58:58.821 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-05-06T03:58:59.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: s= 2024-05-06T03:58:59.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: local status=0 2024-05-06T03:58:59.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-05-06T03:58:59.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-05-06T03:58:59.433 INFO:tasks.workunit.client.0.smithi163.stderr:erasure-code-profile TESTPROFILE does not exist 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: erasure_code_plugin_exists: return 0 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: erasure_code_plugin_exists: local plugin=FAKE 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: local status 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local grepstr 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: local s 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: case `uname` in 2024-05-06T03:58:59.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: uname 2024-05-06T03:58:59.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-05-06T03:58:59.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-05-06T03:58:59.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-05-06T03:58:59.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: local status=5 2024-05-06T03:58:59.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-05-06T03:58:59.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-05-06T03:58:59.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: erasure_code_plugin_exists: return 5 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:59.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:59.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:59.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:59.715 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:59.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:59.716 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:59.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:59.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:59.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:59.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:59.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:59.717 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:59.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:59.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:59.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:59.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:59.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:59.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_expect_failure td/ceph-helpers 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_expect_failure: local dir=td/ceph-helpers 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_expect_failure: setup td/ceph-helpers 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:59.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:59.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:59.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:59.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:59.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:59.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:59.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:59.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:59.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:59.726 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:59.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:59.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:59.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:59.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:59.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:59.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:59.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:59.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:59.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:58:59.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:58:59.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:59.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:59.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/ceph-helpers 2024-05-06T03:58:59.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=FAIL 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/ceph-helpers 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=FAIL 2024-05-06T03:58:59.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:58:59.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:58:59.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-05-06T03:58:59.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: success=true 2024-05-06T03:58:59.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: true 2024-05-06T03:58:59.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: cat td/ceph-helpers/out 2024-05-06T03:58:59.740 INFO:tasks.workunit.client.0.smithi163.stderr:FAIL 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: expect_failure: return 1 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/ceph-helpers 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=FAIL 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-05-06T03:58:59.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: cat td/ceph-helpers/out 2024-05-06T03:58:59.743 INFO:tasks.workunit.client.0.smithi163.stderr:UNEXPECTED 2024-05-06T03:58:59.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: expect_failure: return 1 2024-05-06T03:58:59.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_expect_failure: teardown td/ceph-helpers 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:59.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:59.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:59.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:59.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:59.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:59.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:59.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:59.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:59.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:59.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:59.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:59.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:59.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_flush_pg_stats td/ceph-helpers 2024-05-06T03:58:59.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_flush_pg_stats: local dir=td/ceph-helpers 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_flush_pg_stats: setup td/ceph-helpers 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:58:59.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:58:59.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:58:59.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:58:59.764 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:58:59.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:58:59.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:58:59.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:58:59.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T03:58:59.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:58:59.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:58:59.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:58:59.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:58:59.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:58:59.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:00.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T03:59:00.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:59:00.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:59:00.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:59:00.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:59:00.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:59:00.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:59:00.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:59:00.439 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:59:00.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:59:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-05-06T03:59:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:59:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:59:00.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:59:00.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:59:00.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:59:00.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:59:01.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:59:01.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:59:01.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:59:01.398 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T03:59:01.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:59:01.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:59:01.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1cc5f387-1142-44d2-9ff6-fbf110fe47c9 2024-05-06T03:59:01.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1cc5f387-1142-44d2-9ff6-fbf110fe47c9' 2024-05-06T03:59:01.400 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 1cc5f387-1142-44d2-9ff6-fbf110fe47c9 2024-05-06T03:59:01.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:59:01.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFVThmhSF/GBAAEKG2nnsncuSmWz23YJRlEQ== 2024-05-06T03:59:01.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFVThmhSF/GBAAEKG2nnsncuSmWz23YJRlEQ=="}' 2024-05-06T03:59:01.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1cc5f387-1142-44d2-9ff6-fbf110fe47c9 -i td/ceph-helpers/0/new.json 2024-05-06T03:59:01.692 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:59:01.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T03:59:01.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFVThmhSF/GBAAEKG2nnsncuSmWz23YJRlEQ== --osd-uuid 1cc5f387-1142-44d2-9ff6-fbf110fe47c9 2024-05-06T03:59:01.730 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.727+0000 7f412292f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:01.752 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.748+0000 7f412292f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:01.752 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.749+0000 7f412292f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:01.753 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.750+0000 7f412292f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:59:01.753 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.750+0000 7f412292f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:59:01.753 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.750+0000 7f412292f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:59:01.783 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:01.780+0000 7f412292f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T03:59:07.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T03:59:07.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:59:07.306 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:59:07.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:59:07.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:59:07.874 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:59:07.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:59:07.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T03:59:07.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:59:07.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:59:07.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:59:07.901 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:07.897+0000 7f482d51e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:07.914 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:07.910+0000 7f482d51e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:07.915 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:07.912+0000 7f482d51e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:59:08.336 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:59:08.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:08.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:08.443 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:08.440+0000 7f482d51e640 -1 Falling back to public interface 2024-05-06T03:59:08.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:09.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:09.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:09.772 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:59:09.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:59:09.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:09.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:10.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:10.840 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:10.836+0000 7f482d51e640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:59:11.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:11.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:11.176 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:59:11.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:59:11.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:11.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:11.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:12.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:12.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:12.595 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:59:12.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:59:12.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:12.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:12.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:13.997 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:59:13.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:13.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:13.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:59:13.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:13.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:14.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:15.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:15.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:15.547 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:59:15.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:59:15.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:15.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:15.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:16.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:16.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:16.975 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:59:16.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:59:16.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:16.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:17.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:18.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:18.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:18.538 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T03:59:18.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T03:59:18.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:18.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:18.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:19.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:19.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:19.976 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T03:59:19.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T03:59:19.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:19.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:20.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:21.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:21.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:21.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T03:59:21.543 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T03:59:21.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:21.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:21.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:22.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:22.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:22.949 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T03:59:22.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T03:59:22.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:22.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:23.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:24.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:24.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:24.344 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T03:59:24.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T03:59:24.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:24.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:24.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:25.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:25.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:25.926 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T03:59:25.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T03:59:25.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:25.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:26.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:27.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:27.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:27.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T03:59:27.957 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T03:59:27.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:27.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:28.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:29.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:29.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:29.676 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T03:59:29.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T03:59:29.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:29.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:30.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:31.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:31.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:31.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T03:59:31.766 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T03:59:31.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:31.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:33.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:34.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:34.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:34.041 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T03:59:34.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T03:59:34.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:34.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:34.497 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2826895917,v1:127.0.0.1:6803/2826895917] [v2:127.0.0.1:6804/2826895917,v1:127.0.0.1:6805/2826895917] exists,up 1cc5f387-1142-44d2-9ff6-fbf110fe47c9 2024-05-06T03:59:34.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T03:59:34.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T03:59:34.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T03:59:34.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: test_flush_pg_stats: create_rbd_pool 2024-05-06T03:59:34.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T03:59:34.950 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T03:59:34.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T03:59:34.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T03:59:35.487 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T03:59:35.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T03:59:36.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T03:59:37.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-05-06T03:59:37.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: test_flush_pg_stats: flush_pg_stats 2024-05-06T03:59:37.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T03:59:37.713 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T03:59:38.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T03:59:38.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T03:59:38.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T03:59:38.175 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T03:59:38.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836484 2024-05-06T03:59:38.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836484 2024-05-06T03:59:38.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836484' 2024-05-06T03:59:38.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T03:59:38.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836484 2024-05-06T03:59:38.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T03:59:38.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T03:59:38.300 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836484 2024-05-06T03:59:38.300 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T03:59:38.301 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836484 2024-05-06T03:59:38.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836484 2024-05-06T03:59:38.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-05-06T03:59:38.302 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:59:38.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836482 -lt 21474836484 2024-05-06T03:59:38.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T03:59:39.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T03:59:39.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T03:59:40.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836484 2024-05-06T03:59:40.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-05-06T03:59:40.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: ceph df detail --format=json 2024-05-06T03:59:40.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-05-06T03:59:40.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_flush_pg_stats: stored=6317 2024-05-06T03:59:40.704 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: ceph df detail --format=json 2024-05-06T03:59:40.704 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-05-06T03:59:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_flush_pg_stats: stored_raw=6317 2024-05-06T03:59:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: test_flush_pg_stats: test 6317 -gt 0 2024-05-06T03:59:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: test_flush_pg_stats: test 6317 == 6317 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: test_flush_pg_stats: teardown td/ceph-helpers 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:59:41.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:59:41.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:59:41.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:59:41.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:59:41.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:59:41.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:59:41.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:59:41.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:59:41.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:59:41.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:59:41.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:59:41.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:59:41.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:59:41.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:59:41.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:59:41.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:59:41.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:41.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:41.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_config td/ceph-helpers 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T03:59:41.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T03:59:41.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T03:59:41.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T03:59:41.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T03:59:41.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T03:59:41.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T03:59:41.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T03:59:41.403 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T03:59:41.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:59:41.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T03:59:41.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T03:59:41.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T03:59:41.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T03:59:41.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T03:59:41.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T03:59:41.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T03:59:41.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:41.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:41.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T03:59:41.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T03:59:41.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T03:59:41.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T03:59:41.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T03:59:41.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:41.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:41.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T03:59:41.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:41.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T03:59:42.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T03:59:42.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T03:59:42.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:59:42.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:59:42.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T03:59:42.020 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:59:42.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T03:59:42.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:59:42.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T03:59:42.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-05-06T03:59:42.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T03:59:42.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T03:59:42.183 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T03:59:42.184 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get osd_pool_default_size 2024-05-06T03:59:42.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T03:59:42.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:43.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:59:43.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T03:59:43.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T03:59:43.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T03:59:43.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T03:59:43.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T03:59:43.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e733e85e-df02-44f7-947f-6d1cda3e07b3 2024-05-06T03:59:43.141 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 e733e85e-df02-44f7-947f-6d1cda3e07b3 2024-05-06T03:59:43.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e733e85e-df02-44f7-947f-6d1cda3e07b3' 2024-05-06T03:59:43.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T03:59:43.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvVThmROEpCRAAza8W+kSt90GegK65xGLP8w== 2024-05-06T03:59:43.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvVThmROEpCRAAza8W+kSt90GegK65xGLP8w=="}' 2024-05-06T03:59:43.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e733e85e-df02-44f7-947f-6d1cda3e07b3 -i td/ceph-helpers/0/new.json 2024-05-06T03:59:43.484 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:59:43.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T03:59:43.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQCvVThmROEpCRAAza8W+kSt90GegK65xGLP8w== --osd-uuid e733e85e-df02-44f7-947f-6d1cda3e07b3 2024-05-06T03:59:43.524 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.521+0000 7f7d24210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:43.553 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.550+0000 7f7d24210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:43.554 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.551+0000 7f7d24210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:43.555 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.551+0000 7f7d24210640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:59:43.555 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.551+0000 7f7d24210640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:59:43.555 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.551+0000 7f7d24210640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T03:59:43.585 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:43.581+0000 7f7d24210640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T03:59:49.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T03:59:49.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T03:59:49.057 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T03:59:49.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T03:59:49.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T03:59:49.635 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T03:59:49.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T03:59:49.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-05-06T03:59:49.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T03:59:49.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T03:59:49.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T03:59:49.662 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:49.659+0000 7f0f05096640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:49.664 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:49.661+0000 7f0f05096640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:49.665 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:49.662+0000 7f0f05096640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T03:59:50.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T03:59:50.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T03:59:50.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T03:59:50.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T03:59:50.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T03:59:50.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:50.071 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T03:59:50.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T03:59:50.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:50.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:50.186 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:50.183+0000 7f0f05096640 -1 Falling back to public interface 2024-05-06T03:59:50.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:51.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:51.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:51.506 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T03:59:51.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T03:59:51.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:51.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:51.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:52.559 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T03:59:52.555+0000 7f0f05096640 -1 osd.0 0 log_to_monitors true 2024-05-06T03:59:52.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:52.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:52.918 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T03:59:52.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T03:59:52.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:52.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:54.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:54.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:54.326 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T03:59:54.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T03:59:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:54.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:54.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:55.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:55.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:55.732 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T03:59:55.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T03:59:55.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:55.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:56.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:57.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:57.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:57.151 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T03:59:57.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T03:59:57.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:57.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:57.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T03:59:58.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T03:59:58.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T03:59:58.627 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T03:59:58.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T03:59:58.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T03:59:58.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T03:59:59.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:00.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:00.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:00.041 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:00:00.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:00:00.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:00.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:00.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:01.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:01.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:01.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:00:01.447 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:00:01.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:01.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:01.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:02.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:02.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:02.886 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:00:02.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:00:02.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:02.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:03.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:04.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:04.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:04.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:00:04.309 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:00:04.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:04.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:05.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:06.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:06.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:06.119 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:00:06.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:00:06.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:06.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:07.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:08.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:08.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:08.175 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:00:08.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:00:08.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:08.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:09.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:10.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:10.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:10.117 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:00:10.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:00:10.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:10.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:10.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:11.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:11.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:11.861 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:00:11.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:00:11.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:11.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:12.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:13.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:13.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:13.715 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:00:13.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:00:13.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:13.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:14.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:15.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:15.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:15.618 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:00:15.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:00:15.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:15.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:16.105 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1468509811,v1:127.0.0.1:6803/1468509811] [v2:127.0.0.1:6804/1468509811,v1:127.0.0.1:6805/1468509811] exists,up e733e85e-df02-44f7-947f-6d1cda3e07b3 2024-05-06T04:00:16.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:00:16.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:00:16.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:00:16.107 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-05-06T04:00:16.107 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:00:16.108 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_max_scrubs 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:00:16.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:00:16.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:00:16.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:00:16.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:00:16.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:00:16.300 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:00:16.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:00:16.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:00:16.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:00:16.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:16.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:00:16.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:00:16.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:16.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:00:16.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:00:16.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:00:16.309 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:00:16.309 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:16.309 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:16.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:00:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:00:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:00:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:00:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-05-06T04:00:16.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:00:16.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:00:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:00:16.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:00:16.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:00:16.313 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:00:16.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:00:16.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:00:16.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:00:16.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:16.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:00:16.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:00:16.316 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:16.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:00:16.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:00:16.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:00:16.318 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:00:16.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:16.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:16.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:00:16.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:00:16.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:00:16.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:00:16.320 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:00:16.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:16.321 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:16.321 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:00:16.322 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:00:16.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:00:16.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:00:16.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T04:00:16.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:00:16.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T04:00:17.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:00:17.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:00:17.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:00:17.090 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:17.091 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:00:17.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:00:17.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:00:17.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-05-06T04:00:17.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:00:17.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:00:17.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:00:17.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:00:17.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:00:17.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:00:18.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:00:18.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:00:18.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:00:18.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:00:18.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:18.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:18.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:00:18.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:00:18.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:00:18.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:00:18.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:00:18.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:00:18.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:00:18.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:00:18.395 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:00:18.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:00:18.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:00:18.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:00:18.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:00:18.397 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:00:18.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:18.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:00:18.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:00:18.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:18.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:00:18.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:00:18.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:00:18.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:00:18.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:18.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:18.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:00:18.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:00:18.405 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:00:18.405 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:00:18.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:00:18.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:00:18.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:00:18.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:00:18.406 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:00:18.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:00:18.406 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:00:18.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:00:18.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:00:18.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:00:18.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:18.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:00:18.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:00:18.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:00:18.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:00:18.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:00:18.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:00:18.410 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:00:18.410 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:18.410 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:18.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:00:18.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:00:18.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:00:18.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:00:18.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:00:18.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:18.412 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:18.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:00:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:00:18.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:00:19.068 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:00:19.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:00:19.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:00:19.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:00:19.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:00:20.116 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:00:20.116 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:00:20.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:00:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-05-06T04:00:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:00:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:00:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:00:20.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:00:20.149 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:00:20.150 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:00:20.150 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:00:20.150 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:00:20.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:00:20.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:00:20.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd635a66-0f56-472a-a43a-3f0ec44de9cb 2024-05-06T04:00:20.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bd635a66-0f56-472a-a43a-3f0ec44de9cb' 2024-05-06T04:00:20.152 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 bd635a66-0f56-472a-a43a-3f0ec44de9cb 2024-05-06T04:00:20.152 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:00:20.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUVThmsBK8CRAAV8AvejWUIyp2Ae1dHmqNzw== 2024-05-06T04:00:20.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUVThmsBK8CRAAV8AvejWUIyp2Ae1dHmqNzw=="}' 2024-05-06T04:00:20.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd635a66-0f56-472a-a43a-3f0ec44de9cb -i td/ceph-helpers/0/new.json 2024-05-06T04:00:20.451 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:00:20.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:00:20.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUVThmsBK8CRAAV8AvejWUIyp2Ae1dHmqNzw== --osd-uuid bd635a66-0f56-472a-a43a-3f0ec44de9cb 2024-05-06T04:00:20.489 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.486+0000 7f1798e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:00:20.519 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.516+0000 7f1798e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:00:20.520 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.517+0000 7f1798e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:00:20.521 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.518+0000 7f1798e15640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:00:20.521 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.518+0000 7f1798e15640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:00:20.521 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.518+0000 7f1798e15640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:00:20.560 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:20.556+0000 7f1798e15640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:00:26.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:00:26.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:00:26.087 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:00:26.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:00:26.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:00:26.663 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:00:26.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:00:26.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:00:26.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:00:26.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:00:26.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:00:26.689 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:26.686+0000 7f2a47e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:00:26.702 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:26.699+0000 7f2a47e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:00:26.703 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:26.700+0000 7f2a47e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:00:27.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:00:27.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:00:27.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:00:27.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:00:27.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:00:27.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:27.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:00:27.117 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:00:27.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:27.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:27.234 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:27.231+0000 7f2a47e1d640 -1 Falling back to public interface 2024-05-06T04:00:27.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:28.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:28.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:28.552 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:00:28.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:00:28.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:28.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:28.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:29.605 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:00:29.602+0000 7f2a47e1d640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:00:29.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:29.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:29.994 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:00:29.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:00:29.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:29.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:30.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:31.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:31.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:31.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:00:31.401 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:00:31.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:31.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:31.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:32.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:32.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:32.805 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:00:32.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:00:32.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:32.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:33.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:34.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:34.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:34.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:00:34.212 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:00:34.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:34.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:34.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:35.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:35.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:35.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:00:35.658 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:00:35.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:35.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:36.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:37.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:37.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:37.092 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:00:37.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:00:37.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:37.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:37.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:38.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:38.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:38.557 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:00:38.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:00:38.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:38.558 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:39.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:40.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:40.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:40.008 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:00:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:00:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:40.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:41.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:41.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:41.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:00:41.502 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:00:41.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:41.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:42.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:43.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:43.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:43.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:00:43.066 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:00:43.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:43.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:43.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:44.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:44.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:44.585 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:00:44.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:00:44.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:44.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:45.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:46.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:46.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:46.373 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:00:46.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:00:46.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:46.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:47.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:48.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:48.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:48.376 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:00:48.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:00:48.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:48.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:49.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:50.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:50.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:50.188 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:00:50.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:00:50.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:50.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:51.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:52.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:52.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:52.109 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:00:52.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:00:52.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:52.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:52.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:00:53.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:00:53.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:00:53.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:00:53.695 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:00:53.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:00:53.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:00:54.095 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/688403889,v1:127.0.0.1:6803/688403889] [v2:127.0.0.1:6804/688403889,v1:127.0.0.1:6805/688403889] exists,up bd635a66-0f56-472a-a43a-3f0ec44de9cb 2024-05-06T04:00:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:00:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:00:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:00:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-05-06T04:00:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:00:54.486 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:00:54.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:00:54.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:00:54.982 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:00:54.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:00:55.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:00:56.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-05-06T04:00:56.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:00:56.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:00:56.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:00:56.919 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:00:56.920 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:00:56.920 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:00:56.920 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:00:56.920 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:00:56.920 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:00:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:00:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:00:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:00:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:00:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:00:57.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:00:57.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:00:57.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:00:57.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:00:57.463 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:00:57.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-05-06T04:00:57.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-05-06T04:00:57.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-05-06T04:00:57.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:00:57.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-05-06T04:00:57.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:00:57.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:00:57.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-05-06T04:00:57.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:00:57.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-05-06T04:00:57.581 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836489 2024-05-06T04:00:57.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-05-06T04:00:57.582 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:00:58.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836489 2024-05-06T04:00:58.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:00:59.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:00:59.016 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:00:59.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836489 2024-05-06T04:00:59.433 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:00:59.433 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:00:59.433 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:01:00.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:01:00.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:01:00.041 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:01:00.041 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:01:00.042 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:01:00.042 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:01:00.042 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:01:00.042 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:01:00.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:01:00.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:01:00.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:01:00.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:01:00.960 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:01:01.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-05-06T04:00:54.824856+0000 2024-05-06T04:01:01.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-05-06T04:00:54.824856+0000 2024-05-06T04:01:01.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-05-06T04:01:01.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:01:01.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:01:01.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:01:01.493 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:01:01.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:01:01.495 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:01:01.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:01:01.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:01:01.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:01:01.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:01:01.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:01:01.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:01:01.497 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:01:01.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:01:01.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:01:01.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:01:01.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:01:01.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:01.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:01.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_not_primary td/ceph-helpers 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:01:01.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:01:01.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:01:01.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:01:01.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:01:01.508 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:01:01.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:01:01.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:01:01.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:01:01.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:01:01.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:01:01.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:01:01.511 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:01:01.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:01:01.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:01:01.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:01:01.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:01:01.515 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:01:01.515 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:01.515 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:01.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:01:01.517 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:01:01.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:01:01.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:01:01.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:01:02.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-05-06T04:01:02.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:01:02.211 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:01:02.212 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:01:02.212 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:02.212 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:02.212 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:01:02.212 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:01:02.212 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:01:02.292 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:01:02.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:01:02.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:01:03.217 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:01:03.218 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:01:03.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:01:03.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-05-06T04:01:03.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:01:03.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:01:03.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:01:03.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:01:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:01:03.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:01:03.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:01:03.251 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:01:03.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7660858-0fc4-4cae-881e-8dd6f725119f 2024-05-06T04:01:03.251 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 a7660858-0fc4-4cae-881e-8dd6f725119f 2024-05-06T04:01:03.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a7660858-0fc4-4cae-881e-8dd6f725119f' 2024-05-06T04:01:03.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:01:03.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/VThmX7bFDxAAWNuUT24mU/RFUlY0NLCD/Q== 2024-05-06T04:01:03.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/VThmX7bFDxAAWNuUT24mU/RFUlY0NLCD/Q=="}' 2024-05-06T04:01:03.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7660858-0fc4-4cae-881e-8dd6f725119f -i td/ceph-helpers/0/new.json 2024-05-06T04:01:03.542 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:01:03.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:01:03.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/VThmX7bFDxAAWNuUT24mU/RFUlY0NLCD/Q== --osd-uuid a7660858-0fc4-4cae-881e-8dd6f725119f 2024-05-06T04:01:03.581 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.578+0000 7f30f5985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:03.603 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.600+0000 7f30f5985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:03.604 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.601+0000 7f30f5985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.601+0000 7f30f5985640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:01:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.601+0000 7f30f5985640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:01:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.601+0000 7f30f5985640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:01:03.635 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:03.632+0000 7f30f5985640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:01:09.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:01:09.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:01:09.198 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:01:09.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:01:09.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:01:09.758 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:01:09.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:01:09.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:01:09.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:01:09.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:01:09.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:01:09.784 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:09.781+0000 7f7ab8e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:09.794 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:09.791+0000 7f7ab8e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:09.795 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:09.792+0000 7f7ab8e2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:10.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:01:10.209 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:01:10.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:10.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:10.336 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:10.333+0000 7f7ab8e2c640 -1 Falling back to public interface 2024-05-06T04:01:10.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:11.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:11.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:11.636 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:01:11.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:01:11.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:11.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:12.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:12.680 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:12.677+0000 7f7ab8e2c640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:01:13.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:13.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:13.082 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:01:13.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:01:13.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:13.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:13.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:14.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:14.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:14.719 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:01:14.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:01:14.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:14.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:15.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:16.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:16.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:16.141 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:01:16.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:01:16.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:16.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:16.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:17.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:17.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:17.550 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:01:17.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:01:17.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:17.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:17.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:18.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:18.954 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:01:18.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:18.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:01:18.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:18.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:19.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:20.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:20.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:20.405 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:01:20.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:01:20.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:20.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:20.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:21.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:21.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:21.809 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:01:21.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:01:21.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:21.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:22.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:23.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:23.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:23.243 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:01:23.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:01:23.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:23.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:23.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:24.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:24.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:24.713 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:01:24.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:01:24.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:24.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:25.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:26.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:26.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:26.543 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:01:26.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:01:26.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:26.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:27.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:28.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:28.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:28.271 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:01:28.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:01:28.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:28.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:28.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:29.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:29.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:29.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:01:29.872 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:01:29.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:29.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:30.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:31.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:31.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:31.854 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:01:31.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:01:31.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:31.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:32.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:33.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:33.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:33.656 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:01:33.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:01:33.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:33.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:34.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:35.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:35.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:35.519 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:01:35.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:01:35.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:35.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:01:35.917 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/242671866,v1:127.0.0.1:6803/242671866] [v2:127.0.0.1:6804/242671866,v1:127.0.0.1:6805/242671866] exists,up a7660858-0fc4-4cae-881e-8dd6f725119f 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:01:35.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:01:35.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:01:35.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86356a2d-b045-460a-bc7c-dca6daf229d4 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 86356a2d-b045-460a-bc7c-dca6daf229d4 2024-05-06T04:01:35.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 86356a2d-b045-460a-bc7c-dca6daf229d4' 2024-05-06T04:01:35.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:01:35.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfVjhmRI7INxAAgiuTcCZJSwJyVkPNF1TXLA== 2024-05-06T04:01:35.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfVjhmRI7INxAAgiuTcCZJSwJyVkPNF1TXLA=="}' 2024-05-06T04:01:35.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86356a2d-b045-460a-bc7c-dca6daf229d4 -i td/ceph-helpers/1/new.json 2024-05-06T04:01:36.397 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:01:36.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:01:36.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfVjhmRI7INxAAgiuTcCZJSwJyVkPNF1TXLA== --osd-uuid 86356a2d-b045-460a-bc7c-dca6daf229d4 2024-05-06T04:01:36.438 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.434+0000 7fa0646fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:36.446 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.442+0000 7fa0646fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:36.447 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.443+0000 7fa0646fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:36.447 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.444+0000 7fa0646fb640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:01:36.448 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.444+0000 7fa0646fb640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:01:36.448 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.444+0000 7fa0646fb640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:01:36.495 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:36.491+0000 7fa0646fb640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:01:42.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:01:42.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:01:42.495 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:01:42.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:01:42.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:01:43.048 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:01:43.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:01:43.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:01:43.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:01:43.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:01:43.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:01:43.075 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:43.072+0000 7f5d9e22c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:43.089 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:43.086+0000 7f5d9e22c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:43.090 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:43.087+0000 7f5d9e22c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:01:43.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:01:43.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:01:43.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:01:43.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:01:43.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:01:43.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:43.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:01:43.473 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:01:43.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:43.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:43.625 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:43.622+0000 7f5d9e22c640 -1 Falling back to public interface 2024-05-06T04:01:43.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:44.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:44.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:44.905 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:01:44.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:01:44.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:44.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:45.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:46.009 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:01:46.006+0000 7f5d9e22c640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:01:46.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:46.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:46.310 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:01:46.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:01:46.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:46.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:46.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:47.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:47.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:47.817 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:01:47.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:01:47.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:47.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:48.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:49.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:49.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:49.242 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:01:49.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:01:49.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:49.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:49.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:50.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:50.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:50.648 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:01:50.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:01:50.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:50.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:51.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:52.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:52.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:52.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:01:52.050 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:01:52.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:52.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:52.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:53.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:53.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:53.630 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:01:53.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:01:53.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:53.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:54.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:55.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:55.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:55.167 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:01:55.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:01:55.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:55.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:55.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:56.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:56.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:56.605 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:01:56.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:01:56.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:56.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:58.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:58.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:58.034 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:01:58.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:01:58.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:58.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:01:58.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:01:59.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:01:59.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:01:59.712 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:01:59.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:01:59.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:01:59.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:02:00.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:01.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:01.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:01.797 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:02:01.798 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:02:01.798 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:01.798 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:02:02.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:03.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:03.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:03.543 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:02:03.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:02:03.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:03.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:02:04.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:05.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:05.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:05.522 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:02:05.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:02:05.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:05.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:02:06.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:07.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:07.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:07.241 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:02:07.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:02:07.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:07.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:02:08.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:09.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:09.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:09.223 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:02:09.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:02:09.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:09.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:02:09.696 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4155491621,v1:127.0.0.1:6811/4155491621] [v2:127.0.0.1:6812/4155491621,v1:127.0.0.1:6813/4155491621] exists,up 86356a2d-b045-460a-bc7c-dca6daf229d4 2024-05-06T04:02:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:02:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:02:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:02:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-05-06T04:02:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:02:10.114 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:02:10.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:02:10.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:02:10.641 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:02:10.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:02:11.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:02:13.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-05-06T04:02:13.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:02:13.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:02:13.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:02:13.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:02:13.370 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:02:13.370 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:02:13.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:02:13.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:02:13.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:02:13.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:02:13.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:02:13.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:02:13.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:02:13.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:02:13.460 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:02:13.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:02:13.852 INFO:tasks.workunit.client.0.smithi163.stderr:1' 2024-05-06T04:02:13.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:02:13.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:02:13.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:02:13.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836496 2024-05-06T04:02:13.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836496 2024-05-06T04:02:13.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836496' 2024-05-06T04:02:13.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:02:13.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672966 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672966 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836496 1-42949672966' 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836496 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:02:14.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:02:14.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836496 2024-05-06T04:02:14.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:02:14.085 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836496 2024-05-06T04:02:14.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836496 2024-05-06T04:02:14.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-05-06T04:02:14.085 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:02:14.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836496 2024-05-06T04:02:14.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:02:15.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:02:15.497 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:02:15.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836497 -lt 21474836496 2024-05-06T04:02:15.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:02:15.896 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672966 2024-05-06T04:02:15.896 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:02:15.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:02:15.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672966 2024-05-06T04:02:15.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:02:15.898 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672966 2024-05-06T04:02:15.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672966 2024-05-06T04:02:15.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-05-06T04:02:15.898 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:02:16.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672967 -lt 42949672966 2024-05-06T04:02:16.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:02:16.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:02:16.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:02:16.869 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:02:17.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:02:17.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:02:17.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:02:17.278 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:02:17.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:02:17.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:02:17.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:02:17.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-05-06T04:02:17.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-05-06T04:02:17.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-05-06T04:02:17.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-05-06T04:02:17.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-05-06T04:02:18.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-06T04:02:18.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-05-06T04:02:18.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-05-06T04:02:18.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-05-06T04:02:19.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-05-06T04:02:19.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-05-06T04:02:19.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:02:19.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:02:19.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:02:19.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:02:19.248 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:02:19.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:02:19.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:02:19.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:02:19.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:02:19.251 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:02:19.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:02:19.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:02:19.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:02:19.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:02:19.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:02:19.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:02:19.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:02:19.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:02:19.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:19.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:19.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:02:19.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:02:19.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:02:19.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:02:19.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_num_active_clean td/ceph-helpers 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:02:19.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:02:19.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:02:19.266 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:02:19.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:02:19.266 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:02:19.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:02:19.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:02:19.267 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:02:19.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:02:19.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:02:19.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:02:19.269 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:02:19.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:02:19.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:02:19.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:02:19.271 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:02:19.271 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:19.272 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:19.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:02:19.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:02:19.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:02:19.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:02:19.273 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:02:19.273 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:19.274 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:19.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:02:19.275 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:02:19.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:02:19.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:02:19.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:02:19.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:02:19.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:02:19.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:02:19.746 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:02:19.746 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:19.746 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:19.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:02:19.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:02:19.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:02:19.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:02:19.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:02:19.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:02:19.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:02:19.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:02:19.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:02:20.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:02:20.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:02:20.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:02:20.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-05-06T04:02:20.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:02:20.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:02:20.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:02:20.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:02:20.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:02:20.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:02:20.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:02:20.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:02:20.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:02:20.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:02:20.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:02:21.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6f318ac-8d77-460d-9749-fa7009943e16 2024-05-06T04:02:21.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b6f318ac-8d77-460d-9749-fa7009943e16' 2024-05-06T04:02:21.002 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 b6f318ac-8d77-460d-9749-fa7009943e16 2024-05-06T04:02:21.002 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:02:21.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNVjhmyfnaABAAO0QN92l+OyWojIP2XQka8w== 2024-05-06T04:02:21.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNVjhmyfnaABAAO0QN92l+OyWojIP2XQka8w=="}' 2024-05-06T04:02:21.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6f318ac-8d77-460d-9749-fa7009943e16 -i td/ceph-helpers/0/new.json 2024-05-06T04:02:21.296 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:02:21.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:02:21.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNVjhmyfnaABAAO0QN92l+OyWojIP2XQka8w== --osd-uuid b6f318ac-8d77-460d-9749-fa7009943e16 2024-05-06T04:02:21.335 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.332+0000 7f234f41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:02:21.364 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.362+0000 7f234f41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:02:21.366 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.363+0000 7f234f41f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:02:21.366 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.363+0000 7f234f41f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:02:21.366 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.363+0000 7f234f41f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:02:21.367 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.363+0000 7f234f41f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:02:21.404 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:21.402+0000 7f234f41f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:02:27.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:02:27.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:02:27.238 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:02:27.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:02:27.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:02:27.798 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:02:27.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:02:27.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:02:27.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:02:27.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:02:27.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:02:27.824 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:27.822+0000 7fead8fa7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:02:27.830 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:27.827+0000 7fead8fa7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:02:27.831 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:27.829+0000 7fead8fa7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:02:28.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:02:28.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:02:28.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:02:28.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:02:28.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:02:28.253 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:02:28.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:28.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:02:28.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:28.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:28.360 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:28.358+0000 7fead8fa7640 -1 Falling back to public interface 2024-05-06T04:02:28.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:29.688 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:02:29.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:29.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:02:29.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:29.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:30.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:30.741 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:02:30.738+0000 7fead8fa7640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:02:31.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:31.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:31.114 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:02:31.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:02:31.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:31.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:31.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:32.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:32.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:32.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:02:32.677 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:02:32.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:32.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:33.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:34.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:34.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:34.077 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:02:34.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:02:34.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:34.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:34.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:35.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:35.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:35.535 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:02:35.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:02:35.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:35.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:35.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:36.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:36.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:36.945 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:02:36.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:02:36.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:36.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:37.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:38.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:38.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:38.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:02:38.521 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:02:38.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:38.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:38.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:39.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:39.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:39.944 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:02:39.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:02:39.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:39.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:40.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:41.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:41.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:41.345 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:02:41.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:02:41.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:41.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:41.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:42.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:42.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:42.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:02:42.795 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:02:42.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:42.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:43.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:44.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:44.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:44.888 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:02:44.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:02:44.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:44.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:46.729 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:02:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:02:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:47.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:48.393 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:49.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:50.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:50.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:50.289 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:02:50.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:02:50.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:50.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:51.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:52.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:52.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:52.036 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:02:52.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:02:52.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:52.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:52.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:02:53.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:02:53.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:02:53.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:02:53.692 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:02:53.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:02:53.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:02:54.190 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4189921810,v1:127.0.0.1:6803/4189921810] [v2:127.0.0.1:6804/4189921810,v1:127.0.0.1:6805/4189921810] exists,up b6f318ac-8d77-460d-9749-fa7009943e16 2024-05-06T04:02:54.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:02:54.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:02:54.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:02:54.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-05-06T04:02:54.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:02:54.606 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:02:54.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:02:54.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:02:55.283 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:02:55.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:02:56.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:02:57.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:02:57.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:02:57.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:02:57.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:02:57.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:02:57.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:02:57.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:02:57.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:02:57.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:02:57.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:02:57.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:02:57.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:02:58.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-05-06T04:02:58.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-05-06T04:02:58.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-05-06T04:02:58.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:02:58.065 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-05-06T04:02:58.065 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:02:58.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:02:58.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-05-06T04:02:58.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:02:58.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-05-06T04:02:58.067 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836487 2024-05-06T04:02:58.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-05-06T04:02:58.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:02:58.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-05-06T04:02:58.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:02:59.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:02:59.535 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:02:59.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-05-06T04:02:59.976 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:02:59.977 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:02:59.977 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:03:00.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:03:00.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:03:00.604 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:03:00.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:03:00.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:03:00.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:03:00.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:03:00.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:03:01.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:03:01.036 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:03:01.036 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:03:01.036 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:03:01.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:03:01.637 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:03:01.637 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:03:02.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:03:02.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:03:02.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:03:02.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:03:02.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:03:02.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:03:02.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:03:02.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:03:02.142 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:03:02.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:02.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:03:02.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:03:02.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:02.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:03:02.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:03:02.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:03:02.150 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:03:02.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:02.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:02.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:03:02.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:03:02.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:03:02.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:03:02.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:03:02.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:03:02.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:03:02.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:03:02.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:03:02.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:03:02.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:03:02.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:03:02.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:03:02.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:03:02.156 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:03:02.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:02.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:03:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:03:02.157 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:03:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:03:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:03:02.159 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:03:02.160 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:02.160 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:03:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:03:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:03:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:03:02.162 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:03:02.162 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:02.162 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:02.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:03:02.163 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:03:02.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:02.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:03:02.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:03:02.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:02.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:03:02.840 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:03:02.840 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:03:02.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:03:02.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:03:02.923 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:03:02.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-05-06T04:03:02.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:03:03.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:03:03.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:03:03.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:03:03.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:03:03.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:03.853 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:03:03.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:03:03.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-05-06T04:03:03.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:03:03.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:03:03.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6701db4b-7724-4aa0-9961-b0d8c39bdd66 2024-05-06T04:03:03.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6701db4b-7724-4aa0-9961-b0d8c39bdd66' 2024-05-06T04:03:03.888 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 6701db4b-7724-4aa0-9961-b0d8c39bdd66 2024-05-06T04:03:03.888 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:03:03.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3Vjhm3GqhNRAAsBhhbBw0xDKQglbsJlg72w== 2024-05-06T04:03:03.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3Vjhm3GqhNRAAsBhhbBw0xDKQglbsJlg72w=="}' 2024-05-06T04:03:03.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6701db4b-7724-4aa0-9961-b0d8c39bdd66 -i td/ceph-helpers/0/new.json 2024-05-06T04:03:04.172 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:03:04.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:03:04.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3Vjhm3GqhNRAAsBhhbBw0xDKQglbsJlg72w== --osd-uuid 6701db4b-7724-4aa0-9961-b0d8c39bdd66 2024-05-06T04:03:04.212 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.209+0000 7f39d64b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:04.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.237+0000 7f39d64b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:04.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.239+0000 7f39d64b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:04.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.239+0000 7f39d64b5640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:03:04.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.239+0000 7f39d64b5640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:03:04.243 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.239+0000 7f39d64b5640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:03:04.280 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:04.277+0000 7f39d64b5640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:03:09.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:03:09.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:03:09.786 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:03:09.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:03:09.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:03:10.356 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:03:10.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:03:10.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:03:10.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:03:10.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:03:10.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:03:10.382 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:10.379+0000 7f495f20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:10.383 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:10.381+0000 7f495f20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:10.384 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:10.382+0000 7f495f20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:10.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:03:10.800 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:03:10.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:10.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:10.906 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:10.904+0000 7f495f20c640 -1 Falling back to public interface 2024-05-06T04:03:11.233 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:12.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:12.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:12.235 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:03:12.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:03:12.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:12.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:12.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:13.278 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:13.276+0000 7f495f20c640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:03:13.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:13.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:13.669 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:03:13.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:03:13.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:13.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:14.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:15.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:15.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:15.332 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:03:15.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:03:15.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:15.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:15.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:16.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:16.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:16.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:03:16.763 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:03:16.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:16.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:17.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:18.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:18.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:18.173 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:03:18.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:03:18.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:18.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:18.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:19.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:19.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:19.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:03:19.581 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:03:19.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:19.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:20.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:21.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:21.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:21.014 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:03:21.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:03:21.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:21.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:21.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:22.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:22.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:22.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:03:22.419 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:03:22.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:22.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:22.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:23.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:23.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:23.830 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:03:23.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:03:23.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:23.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:24.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:25.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:25.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:25.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:03:25.347 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:03:25.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:25.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:25.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:26.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:26.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:26.848 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:03:26.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:03:26.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:26.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:27.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:28.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:28.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:28.538 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:03:28.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:03:28.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:28.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:29.329 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:30.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:30.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:30.330 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:03:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:03:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:31.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:32.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:32.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:32.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:03:32.127 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:03:32.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:32.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:32.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:33.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:33.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:33.593 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:03:33.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:03:33.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:33.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:34.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:35.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:35.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:35.222 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:03:35.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:03:35.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:35.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:35.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:36.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:36.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:36.812 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:03:36.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:03:36.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:36.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:37.242 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/234363305,v1:127.0.0.1:6803/234363305] [v2:127.0.0.1:6804/234363305,v1:127.0.0.1:6805/234363305] exists,up 6701db4b-7724-4aa0-9961-b0d8c39bdd66 2024-05-06T04:03:37.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:03:37.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:03:37.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:03:37.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-05-06T04:03:37.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:03:37.657 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:03:37.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:03:37.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:03:38.168 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:03:38.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:03:39.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:03:40.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-05-06T04:03:40.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:03:40.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:03:40.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:03:40.245 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:03:40.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:03:40.246 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:03:40.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:03:40.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:03:40.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:03:40.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:03:40.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:03:40.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:03:40.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:03:40.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:03:40.336 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:03:40.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:03:40.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:03:40.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:03:40.754 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:03:40.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-05-06T04:03:40.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-05-06T04:03:40.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-05-06T04:03:40.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:03:40.870 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-05-06T04:03:40.870 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:03:40.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:03:40.871 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-05-06T04:03:40.871 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:03:40.872 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836487 2024-05-06T04:03:40.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-05-06T04:03:40.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-05-06T04:03:40.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:03:41.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-05-06T04:03:41.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:03:42.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:03:42.324 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:03:42.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836487 2024-05-06T04:03:42.765 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:03:42.765 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:03:42.765 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:03:43.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:03:43.317 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:03:43.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:03:43.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:03:43.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:03:43.698 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-05-06T04:03:44.241 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-05-06T04:03:44.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:03:44.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:03:44.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:03:44.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:03:44.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:03:44.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:03:44.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:03:44.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:03:44.748 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:03:44.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:44.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:03:44.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:03:44.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:44.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:03:44.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:03:44.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:03:44.756 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:03:44.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:44.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:44.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_num_pgs td/ceph-helpers 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:03:44.759 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:03:44.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:03:44.760 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:03:44.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:03:44.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:03:44.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:03:44.760 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:03:44.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:03:44.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:03:44.761 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:03:44.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:03:44.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:03:44.762 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:03:44.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:44.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:03:44.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:03:44.763 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:03:44.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:03:44.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:03:44.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:03:44.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:03:44.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:44.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:44.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:03:44.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:03:44.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:03:44.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:03:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:03:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:03:44.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:03:44.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:45.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:03:45.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:03:45.441 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:03:45.442 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:45.524 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:45.525 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:03:45.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:03:45.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:03:45.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-05-06T04:03:45.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:03:45.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:03:45.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:03:45.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:03:45.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:03:45.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:46.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:46.457 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:03:46.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:03:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:03:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:03:46.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:03:46.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:03:46.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78099ef0-ad1f-49f4-87e1-a7a619be2419 2024-05-06T04:03:46.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 78099ef0-ad1f-49f4-87e1-a7a619be2419' 2024-05-06T04:03:46.490 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 78099ef0-ad1f-49f4-87e1-a7a619be2419 2024-05-06T04:03:46.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:03:46.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiVjhmfqX1HRAAQYgBhfhAvaFi+7F/qVYwBA== 2024-05-06T04:03:46.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiVjhmfqX1HRAAQYgBhfhAvaFi+7F/qVYwBA=="}' 2024-05-06T04:03:46.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78099ef0-ad1f-49f4-87e1-a7a619be2419 -i td/ceph-helpers/0/new.json 2024-05-06T04:03:46.781 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:03:46.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:03:46.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiVjhmfqX1HRAAQYgBhfhAvaFi+7F/qVYwBA== --osd-uuid 78099ef0-ad1f-49f4-87e1-a7a619be2419 2024-05-06T04:03:46.820 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.817+0000 7f92aa40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:46.849 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.846+0000 7f92aa40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:46.850 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.847+0000 7f92aa40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:46.851 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.848+0000 7f92aa40d640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:03:46.851 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.848+0000 7f92aa40d640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:03:46.851 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.848+0000 7f92aa40d640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:03:46.889 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:46.887+0000 7f92aa40d640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:03:52.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:03:52.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:03:52.412 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:03:52.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:03:52.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:03:52.982 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:03:52.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:03:52.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:03:52.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:03:52.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:03:52.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:03:53.008 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:53.005+0000 7f5e82014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:53.037 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:53.034+0000 7f5e82014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:53.038 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:53.035+0000 7f5e82014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:03:53.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:03:53.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:03:53.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:03:53.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:03:53.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:03:53.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:53.468 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:03:53.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:03:53.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:53.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:53.580 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:53.578+0000 7f5e82014640 -1 Falling back to public interface 2024-05-06T04:03:53.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:54.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:54.894 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:03:54.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:54.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:03:54.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:54.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:55.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:55.921 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:03:55.918+0000 7f5e82014640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:03:56.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:56.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:56.337 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:03:56.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:03:56.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:56.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:56.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:57.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:57.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:57.992 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:03:57.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:03:57.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:57.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:58.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:03:59.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:03:59.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:03:59.398 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:03:59.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:03:59.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:03:59.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:03:59.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:00.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:00.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:00.806 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:04:00.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:04:00.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:00.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:01.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:02.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:02.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:02.222 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:04:02.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:04:02.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:02.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:02.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:03.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:03.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:03.675 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:04:03.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:04:03.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:03.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:04.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:05.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:05.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:05.080 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:04:05.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:04:05.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:05.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:05.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:06.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:06.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:06.489 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:04:06.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:04:06.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:06.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:07.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:08.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:08.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:08.165 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:04:08.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:04:08.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:08.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:08.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:09.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:09.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:09.696 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:04:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:04:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:09.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:10.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:11.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:11.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:11.143 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:04:11.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:04:11.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:11.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:11.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:12.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:12.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:12.870 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:04:12.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:04:12.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:12.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:13.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:14.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:14.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:14.651 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:04:14.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:04:14.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:14.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:15.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:16.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:16.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:16.058 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:04:16.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:04:16.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:16.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:16.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:17.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:17.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:17.836 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:04:17.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:04:17.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:17.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:18.331 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/513644028,v1:127.0.0.1:6803/513644028] [v2:127.0.0.1:6804/513644028,v1:127.0.0.1:6805/513644028] exists,up 78099ef0-ad1f-49f4-87e1-a7a619be2419 2024-05-06T04:04:18.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:04:18.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:04:18.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:04:18.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-05-06T04:04:18.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:04:18.789 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:04:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:04:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:04:19.507 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:04:19.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:04:20.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:04:21.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-05-06T04:04:21.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:04:21.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:04:21.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:04:21.628 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:04:21.629 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:04:21.629 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:04:21.629 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:04:21.629 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:04:21.629 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:04:21.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:04:21.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:04:21.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:04:21.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:04:21.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:04:21.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:04:22.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:04:22.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:04:22.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:04:22.146 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:04:22.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836484 2024-05-06T04:04:22.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836484 2024-05-06T04:04:22.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836484' 2024-05-06T04:04:22.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:04:22.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836484 2024-05-06T04:04:22.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:04:22.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:04:22.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836484 2024-05-06T04:04:22.264 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:04:22.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836484 2024-05-06T04:04:22.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-05-06T04:04:22.265 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836484 2024-05-06T04:04:22.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:04:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836482 -lt 21474836484 2024-05-06T04:04:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:04:23.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:04:23.714 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:04:24.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836484 2024-05-06T04:04:24.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:04:24.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:04:24.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:04:24.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:04:25.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:04:25.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:04:25.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:04:25.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:04:25.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:04:25.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:04:25.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:04:25.620 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-05-06T04:04:25.620 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:04:25.620 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:04:26.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:04:26.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:04:26.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:04:26.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:04:26.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:04:26.355 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:04:26.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:04:26.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:04:26.356 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:04:26.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:04:26.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:04:26.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:04:26.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:04:26.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:04:26.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:04:26.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:04:26.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:04:26.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:26.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:26.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:04:26.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:04:26.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:04:26.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:04:26.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_op_scheduler td/ceph-helpers 2024-05-06T04:04:26.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: test_get_op_scheduler: local dir=td/ceph-helpers 2024-05-06T04:04:26.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: test_get_op_scheduler: setup td/ceph-helpers 2024-05-06T04:04:26.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:04:26.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:04:26.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:04:26.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:04:26.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:04:26.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:04:26.372 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:04:26.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:04:26.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:04:26.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:04:26.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:04:26.372 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:04:26.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:04:26.373 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:04:26.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:04:26.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:04:26.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:04:26.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:04:26.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:04:26.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:04:26.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:04:26.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:04:26.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:04:26.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:04:26.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:04:26.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:26.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:26.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:04:26.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:04:26.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:04:26.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:04:26.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:04:26.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:26.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:26.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:04:26.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:04:26.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:04:26.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:04:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:04:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-05-06T04:04:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:04:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:04:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:04:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:04:26.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:04:26.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T04:04:26.840 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:04:26.840 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:04:26.841 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:04:26.841 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:04:26.841 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:26.841 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:26.841 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:04:26.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T04:04:27.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:04:27.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:04:27.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:04:27.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:04:27.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:04:27.131 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:04:27.132 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:27.132 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:27.132 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:04:27.132 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:04:27.132 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:04:27.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-05-06T04:04:27.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:04:27.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:04:27.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:04:27.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:04:27.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:04:27.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:04:28.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:04:28.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:04:28.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:04:28.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df7c849c-e06d-4afe-b57d-41cd8ee74c63 2024-05-06T04:04:28.088 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 df7c849c-e06d-4afe-b57d-41cd8ee74c63 2024-05-06T04:04:28.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 df7c849c-e06d-4afe-b57d-41cd8ee74c63' 2024-05-06T04:04:28.088 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:04:28.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMVjhmJiQDBhAA9O5DcHBzKsbRVDdmNrbLMw== 2024-05-06T04:04:28.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMVjhmJiQDBhAA9O5DcHBzKsbRVDdmNrbLMw=="}' 2024-05-06T04:04:28.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df7c849c-e06d-4afe-b57d-41cd8ee74c63 -i td/ceph-helpers/0/new.json 2024-05-06T04:04:28.382 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:04:28.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:04:28.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQDMVjhmJiQDBhAA9O5DcHBzKsbRVDdmNrbLMw== --osd-uuid df7c849c-e06d-4afe-b57d-41cd8ee74c63 2024-05-06T04:04:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.418+0000 7f6cc642e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:28.450 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.446+0000 7f6cc642e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:28.451 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.448+0000 7f6cc642e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:28.452 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.448+0000 7f6cc642e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:04:28.452 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.448+0000 7f6cc642e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:04:28.452 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.448+0000 7f6cc642e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:04:28.490 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:28.487+0000 7f6cc642e640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:04:34.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:04:34.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:04:34.006 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:04:34.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:04:34.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:04:34.595 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:04:34.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:04:34.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-05-06T04:04:34.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:04:34.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:04:34.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:04:34.622 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:34.619+0000 7f66b3c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:34.624 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:34.621+0000 7f66b3c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:34.625 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:34.622+0000 7f66b3c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:35.029 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:04:35.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:04:35.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:35.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:35.146 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:35.144+0000 7f66b3c14640 -1 Falling back to public interface 2024-05-06T04:04:35.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:36.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:36.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:36.456 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:04:36.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:04:36.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:36.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:36.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:37.491 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:37.489+0000 7f66b3c14640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:04:37.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:37.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:37.890 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:04:37.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:04:37.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:37.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:04:38.345 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2385362858,v1:127.0.0.1:6803/2385362858] [v2:127.0.0.1:6804/2385362858,v1:127.0.0.1:6805/2385362858] exists,up df7c849c-e06d-4afe-b57d-41cd8ee74c63 2024-05-06T04:04:38.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:04:38.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:04:38.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:04:38.347 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_get_op_scheduler: get_op_scheduler 0 2024-05-06T04:04:38.347 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: get_op_scheduler: local id=0 2024-05-06T04:04:38.347 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: get_op_scheduler: get_config osd 0 osd_op_queue 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:04:38.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_op_queue 2024-05-06T04:04:38.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_get_op_scheduler: test wpq = wpq 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:04:38.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:04:38.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04a36eb2-f436-463d-8608-9d975e446df4 2024-05-06T04:04:38.439 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 04a36eb2-f436-463d-8608-9d975e446df4 2024-05-06T04:04:38.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 04a36eb2-f436-463d-8608-9d975e446df4' 2024-05-06T04:04:38.440 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:04:38.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWVjhmPd3uGhAAFZyhVW4y1YOGi5To1x1C3w== 2024-05-06T04:04:38.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWVjhmPd3uGhAAFZyhVW4y1YOGi5To1x1C3w=="}' 2024-05-06T04:04:38.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04a36eb2-f436-463d-8608-9d975e446df4 -i td/ceph-helpers/1/new.json 2024-05-06T04:04:38.924 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:04:38.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:04:38.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDWVjhmPd3uGhAAFZyhVW4y1YOGi5To1x1C3w== --osd-uuid 04a36eb2-f436-463d-8608-9d975e446df4 2024-05-06T04:04:38.965 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:38.962+0000 7fbba9bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:39.011 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:39.008+0000 7fbba9bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:39.012 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:39.009+0000 7fbba9bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:39.012 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:39.010+0000 7fbba9bdc640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:04:39.012 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:39.010+0000 7fbba9bdc640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:04:39.012 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:39.010+0000 7fbba9bdc640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:04:39.108 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:39.106+0000 7fbba9bdc640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:04:44.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:04:44.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:04:44.930 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:04:44.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:04:44.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:04:45.483 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:04:45.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:04:45.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-05-06T04:04:45.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:04:45.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:04:45.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:04:45.510 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:45.507+0000 7fb1a01fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:45.517 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:45.514+0000 7fb1a01fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:45.518 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:45.515+0000 7fb1a01fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:04:45.940 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:04:45.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:45.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:46.047 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:46.045+0000 7fb1a01fc640 -1 Falling back to public interface 2024-05-06T04:04:46.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:47.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:47.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:47.331 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:04:47.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:04:47.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:47.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:47.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:48.428 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:04:48.425+0000 7fb1a01fc640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:04:48.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:48.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:48.769 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:04:48.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:04:48.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:48.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:49.219 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:50.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:50.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:50.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:04:50.221 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:04:50.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:50.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:50.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:51.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:51.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:51.629 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:04:51.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:04:51.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:51.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:52.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:53.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:53.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:53.033 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:04:53.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:04:53.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:53.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:53.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:54.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:54.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:54.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:04:54.568 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:04:54.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:54.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:54.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:55.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:55.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:55.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:04:55.978 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:04:55.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:55.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:56.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:57.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:57.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:57.385 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:04:57.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:04:57.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:57.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:57.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:04:58.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:04:58.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:04:58.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:04:58.791 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:04:58.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:04:58.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:04:59.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:00.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:00.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:00.291 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:05:00.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:05:00.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:00.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:01.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:02.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:02.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:02.646 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:05:02.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:05:02.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:02.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:03.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:04.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:04.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:04.182 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:05:04.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:05:04.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:04.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:04.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:05.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:05.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:05.611 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:05:05.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:05:05.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:05.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:06.444 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:07.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:07.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:07.446 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:05:07.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:05:07.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:07.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:08.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:09.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:09.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:09.156 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:05:09.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:05:09.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:09.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:09.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:10.993 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:05:10.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:05:10.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:10.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:05:11.529 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3646855784,v1:127.0.0.1:6811/3646855784] [v2:127.0.0.1:6812/3646855784,v1:127.0.0.1:6813/3646855784] exists,up 04a36eb2-f436-463d-8608-9d975e446df4 2024-05-06T04:05:11.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_get_op_scheduler: get_op_scheduler 1 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: get_op_scheduler: local id=1 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: get_op_scheduler: get_config osd 1 osd_op_queue 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-06T04:05:11.532 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:05:11.533 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:11.533 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:11.533 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.1.asok 2024-05-06T04:05:11.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:05:11.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-osd.1.asok config get osd_op_queue 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_get_op_scheduler: teardown td/ceph-helpers 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:05:11.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:05:11.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:05:11.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:05:11.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:05:11.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:05:11.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:05:11.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:05:11.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:05:11.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:05:11.726 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:05:11.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:11.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:05:11.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:05:11.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:11.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:05:11.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:05:11.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:05:11.735 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:05:11.735 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:11.735 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:11.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:05:11.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:05:11.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:05:11.738 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:05:11.738 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:05:11.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:05:11.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:05:11.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:05:11.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:05:11.740 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:05:11.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:05:11.740 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:05:11.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:05:11.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:05:11.741 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:05:11.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:11.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:05:11.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:05:11.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:11.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:05:11.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:05:11.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:05:11.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:05:11.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:11.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:11.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:05:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:05:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:05:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:05:11.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:05:11.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:11.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:11.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:05:11.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:05:11.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:05:11.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:05:11.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:05:11.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:05:11.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:05:11.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:05:11.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:05:11.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:05:11.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:05:11.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:12.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:05:12.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:05:12.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:05:12.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:05:12.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:05:12.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:05:12.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:05:12.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:05:12.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:05:12.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:12.493 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:12.494 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:05:12.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:05:12.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:05:12.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:13.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:13.414 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:05:13.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:05:13.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-05-06T04:05:13.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:05:13.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:05:13.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:05:13.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:05:13.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7b04ad5-316e-48f5-9b90-8d43f1996b53 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d7b04ad5-316e-48f5-9b90-8d43f1996b53' 2024-05-06T04:05:13.448 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 d7b04ad5-316e-48f5-9b90-8d43f1996b53 2024-05-06T04:05:13.449 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:05:13.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5VjhmvCh8GxAAbT9j+Az3HeQgcNjujmNwiw== 2024-05-06T04:05:13.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5VjhmvCh8GxAAbT9j+Az3HeQgcNjujmNwiw=="}' 2024-05-06T04:05:13.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7b04ad5-316e-48f5-9b90-8d43f1996b53 -i td/ceph-helpers/0/new.json 2024-05-06T04:05:13.731 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:05:13.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:05:13.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5VjhmvCh8GxAAbT9j+Az3HeQgcNjujmNwiw== --osd-uuid d7b04ad5-316e-48f5-9b90-8d43f1996b53 2024-05-06T04:05:13.770 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.767+0000 7fb45371e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:13.800 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.797+0000 7fb45371e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:13.801 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.798+0000 7fb45371e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:13.801 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.799+0000 7fb45371e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:05:13.801 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.799+0000 7fb45371e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:05:13.802 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.799+0000 7fb45371e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:05:13.840 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:13.837+0000 7fb45371e640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:05:19.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:05:19.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:05:19.490 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:05:19.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:05:19.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:05:20.041 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:05:20.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:05:20.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:05:20.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:05:20.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:05:20.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:05:20.067 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:20.065+0000 7f194dcd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:20.072 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:20.069+0000 7f194dcd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:20.073 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:20.071+0000 7f194dcd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:20.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:05:20.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:05:20.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:20.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:20.602 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:20.600+0000 7f194dcd5640 -1 Falling back to public interface 2024-05-06T04:05:20.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:21.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:21.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:21.888 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:05:21.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:05:21.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:21.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:22.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:22.940 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:22.938+0000 7f194dcd5640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:05:23.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:23.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:23.314 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:05:23.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:05:23.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:23.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:23.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:24.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:24.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:24.718 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:05:24.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:05:24.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:24.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:25.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:26.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:26.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:26.122 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:05:26.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:05:26.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:26.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:26.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:27.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:27.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:27.718 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:05:27.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:05:27.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:27.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:28.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:29.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:29.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:29.125 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:05:29.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:05:29.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:29.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:29.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:30.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:30.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:30.669 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:05:30.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:05:30.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:30.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:31.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:32.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:32.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:32.115 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:05:32.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:05:32.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:32.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:32.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:33.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:33.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:33.523 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:05:33.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:05:33.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:33.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:34.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:35.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:35.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:35.177 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:05:35.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:05:35.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:35.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:35.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:36.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:36.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:36.810 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:05:36.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:05:36.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:36.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:37.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:38.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:38.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:38.312 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:05:38.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:05:38.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:38.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:39.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:40.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:40.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:40.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:05:40.062 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:05:40.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:40.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:41.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:42.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:42.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:42.064 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:05:42.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:05:42.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:42.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:43.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:44.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:44.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:44.174 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:05:44.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:05:44.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:44.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:45.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:05:46.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:05:46.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:05:46.146 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:05:46.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:05:46.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:05:46.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:05:46.598 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3177587736,v1:127.0.0.1:6803/3177587736] [v2:127.0.0.1:6804/3177587736,v1:127.0.0.1:6805/3177587736] exists,up d7b04ad5-316e-48f5-9b90-8d43f1996b53 2024-05-06T04:05:46.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:05:46.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:05:46.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:05:46.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-05-06T04:05:46.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:05:47.029 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:05:47.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:05:47.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:05:47.522 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:05:47.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:05:48.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:05:49.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-05-06T04:05:49.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:05:49.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:05:49.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:05:49.543 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:05:49.544 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:05:49.544 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:05:49.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:05:49.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:05:49.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:05:49.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:05:49.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:05:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:05:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:05:49.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:05:49.634 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:05:50.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:05:50.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:05:50.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:05:50.053 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:05:50.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836486 2024-05-06T04:05:50.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836486 2024-05-06T04:05:50.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836486' 2024-05-06T04:05:50.169 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:05:50.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836486 2024-05-06T04:05:50.169 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:05:50.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:05:50.170 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-05-06T04:05:50.170 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:05:50.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836486 2024-05-06T04:05:50.171 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836486 2024-05-06T04:05:50.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-05-06T04:05:50.172 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:05:50.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836486 2024-05-06T04:05:50.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:05:51.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:05:51.597 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:05:52.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836486 2024-05-06T04:05:52.038 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:05:52.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:05:52.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:05:52.591 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:05:52.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:05:52.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:05:52.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:05:52.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:05:53.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:05:53.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:05:53.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:05:53.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-05-06T04:05:53.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-05-06T04:05:53.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-05-06T04:05:53.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-05-06T04:05:53.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-05-06T04:05:53.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:05:53.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:05:54.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:05:54.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:05:54.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:05:54.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:05:54.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:05:54.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:05:54.033 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:05:54.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:54.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:05:54.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:05:54.034 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:54.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:05:54.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:05:54.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:05:54.041 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:05:54.041 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:54.041 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:54.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:05:54.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:05:54.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:05:54.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:05:54.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_osds td/ceph-helpers 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:05:54.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:05:54.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:05:54.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:05:54.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:05:54.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:05:54.046 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:05:54.046 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:05:54.046 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:05:54.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:54.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:05:54.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:05:54.048 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:05:54.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:05:54.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:05:54.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:05:54.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:05:54.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:54.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:54.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:05:54.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:05:54.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:05:54.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:05:54.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:05:54.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:54.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:54.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:05:54.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:05:54.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:05:54.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:05:54.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:05:54.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-05-06T04:05:54.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:05:54.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:05:54.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:05:54.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:05:54.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:05:54.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-05-06T04:05:54.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:05:54.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:05:54.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:05:54.514 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:05:54.515 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:54.515 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:54.515 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:54.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-05-06T04:05:54.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:05:54.700 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:54.701 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:54.701 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:05:54.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:05:54.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:05:54.780 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:54.781 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:54.781 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:05:54.781 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:05:54.781 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:05:54.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-05-06T04:05:54.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:05:54.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:05:54.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:05:54.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:05:54.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:05:54.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:55.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:05:55.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:05:55.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-05-06T04:05:55.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:05:55.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:05:55.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:05:55.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:05:55.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:05:55.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:05:55.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:05:55.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:05:55.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:05:55.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:05:55.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:05:55.748 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:05:55.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a07dc69e-40a9-4a85-b94a-0abd345635f8 2024-05-06T04:05:55.748 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 a07dc69e-40a9-4a85-b94a-0abd345635f8 2024-05-06T04:05:55.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a07dc69e-40a9-4a85-b94a-0abd345635f8' 2024-05-06T04:05:55.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:05:55.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjVzhmjvE1LRAA9J5WZRK/zJVkWjwXcjgLnQ== 2024-05-06T04:05:55.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjVzhmjvE1LRAA9J5WZRK/zJVkWjwXcjgLnQ=="}' 2024-05-06T04:05:55.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a07dc69e-40a9-4a85-b94a-0abd345635f8 -i td/ceph-helpers/0/new.json 2024-05-06T04:05:56.047 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:05:56.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:05:56.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjVzhmjvE1LRAA9J5WZRK/zJVkWjwXcjgLnQ== --osd-uuid a07dc69e-40a9-4a85-b94a-0abd345635f8 2024-05-06T04:05:56.091 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.089+0000 7f7c8640c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:56.116 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.113+0000 7f7c8640c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:56.117 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.115+0000 7f7c8640c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:05:56.118 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.115+0000 7f7c8640c640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:05:56.118 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.115+0000 7f7c8640c640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:05:56.118 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.115+0000 7f7c8640c640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:05:56.156 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:05:56.153+0000 7f7c8640c640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:06:01.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:06:01.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:06:01.680 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:06:01.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:06:01.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:06:02.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:06:02.248 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:06:02.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:06:02.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:06:02.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:06:02.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:06:02.274 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:02.271+0000 7feae14ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:02.287 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:02.285+0000 7feae14ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:02.288 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:02.286+0000 7feae14ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:02.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:06:02.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:06:02.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:06:02.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:06:02.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:06:02.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:02.711 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:06:02.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:06:02.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:02.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:02.822 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:02.820+0000 7feae14ff640 -1 Falling back to public interface 2024-05-06T04:06:03.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:04.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:04.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:04.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:06:04.137 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:06:04.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:04.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:04.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:05.199 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:05.196+0000 7feae14ff640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:06:05.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:05.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:05.588 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:06:05.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:06:05.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:05.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:06.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:07.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:07.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:07.136 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:06:07.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:06:07.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:07.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:07.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:08.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:08.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:08.590 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:06:08.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:06:08.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:08.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:09.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:10.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:10.106 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:06:10.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:10.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:06:10.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:10.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:10.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:11.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:11.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:11.519 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:06:11.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:06:11.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:11.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:11.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:12.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:12.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:12.959 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:06:12.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:06:12.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:12.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:13.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:14.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:14.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:14.371 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:06:14.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:06:14.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:14.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:14.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:15.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:15.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:15.774 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:06:15.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:06:15.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:15.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:16.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:17.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:17.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:17.185 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:06:17.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:06:17.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:17.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:17.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:18.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:18.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:18.612 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:06:18.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:06:18.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:18.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:19.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:20.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:20.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:20.158 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:06:20.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:06:20.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:20.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:20.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:21.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:21.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:21.905 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:06:21.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:06:21.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:21.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:22.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:23.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:23.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:23.416 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:06:23.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:06:23.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:23.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:24.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:25.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:25.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:25.134 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:06:25.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:06:25.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:25.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:26.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:27.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:27.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:27.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:06:27.079 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:06:27.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:27.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:27.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:28.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:28.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:28.704 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:06:28.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:06:28.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:28.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:06:29.098 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/762034652,v1:127.0.0.1:6803/762034652] [v2:127.0.0.1:6804/762034652,v1:127.0.0.1:6805/762034652] exists,up a07dc69e-40a9-4a85-b94a-0abd345635f8 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:06:29.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:06:29.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:06:29.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd0f5ce7-b423-40d8-b158-b015cd7d50c1 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cd0f5ce7-b423-40d8-b158-b015cd7d50c1' 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 cd0f5ce7-b423-40d8-b158-b015cd7d50c1 2024-05-06T04:06:29.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:06:29.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFVzhmMfryBhAAYCCU3+G+sbKGYC4ZiQR3Vg== 2024-05-06T04:06:29.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFVzhmMfryBhAAYCCU3+G+sbKGYC4ZiQR3Vg=="}' 2024-05-06T04:06:29.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd0f5ce7-b423-40d8-b158-b015cd7d50c1 -i td/ceph-helpers/1/new.json 2024-05-06T04:06:29.641 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:06:29.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:06:29.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFVzhmMfryBhAAYCCU3+G+sbKGYC4ZiQR3Vg== --osd-uuid cd0f5ce7-b423-40d8-b158-b015cd7d50c1 2024-05-06T04:06:29.681 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.678+0000 7fc5d4987640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:29.682 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.680+0000 7fc5d4987640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:29.683 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.681+0000 7fc5d4987640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:29.684 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.681+0000 7fc5d4987640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:06:29.684 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.681+0000 7fc5d4987640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:06:29.684 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.681+0000 7fc5d4987640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:06:29.756 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:29.754+0000 7fc5d4987640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:06:35.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:06:35.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:06:35.804 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:06:35.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:06:35.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:06:36.373 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:06:36.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:06:36.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:06:36.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:06:36.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:06:36.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:06:36.398 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:36.395+0000 7f0f1b544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:36.412 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:36.408+0000 7f0f1b544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:36.413 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:36.410+0000 7f0f1b544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:36.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:06:36.827 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:06:36.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:36.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:36.944 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:36.940+0000 7f0f1b544640 -1 Falling back to public interface 2024-05-06T04:06:37.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:38.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:38.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:38.261 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:06:38.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:06:38.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:38.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:38.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:39.398 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:06:39.396+0000 7f0f1b544640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:06:39.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:39.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:39.696 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:06:39.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:06:39.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:39.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:40.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:41.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:41.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:41.251 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:06:41.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:06:41.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:41.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:41.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:42.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:42.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:42.791 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:06:42.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:06:42.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:42.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:43.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:44.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:44.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:44.201 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:06:44.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:06:44.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:44.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:44.626 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:45.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:45.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:45.628 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:06:45.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:06:45.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:45.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:46.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:47.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:47.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:47.109 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:06:47.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:06:47.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:47.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:47.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:48.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:48.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:48.615 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:06:48.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:06:48.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:48.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:49.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:50.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:50.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:50.056 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:06:50.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:06:50.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:50.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:50.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:51.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:51.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:51.569 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:06:51.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:06:51.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:51.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:51.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:52.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:52.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:52.981 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:06:52.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:06:52.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:52.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:53.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:54.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:54.734 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:06:54.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:54.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:06:54.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:54.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:55.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:56.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:56.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:56.503 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:06:56.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:06:56.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:56.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:57.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:06:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:06:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:06:58.700 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:06:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:06:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:06:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:06:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:00.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:00.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:00.561 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:07:00.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:07:00.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:00.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:07:01.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:02.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:02.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:02.243 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:07:02.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:07:02.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:02.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:07:02.660 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3251627519,v1:127.0.0.1:6811/3251627519] [v2:127.0.0.1:6812/3251627519,v1:127.0.0.1:6813/3251627519] exists,up cd0f5ce7-b423-40d8-b158-b015cd7d50c1 2024-05-06T04:07:02.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:07:02.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:07:02.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:07:02.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-05-06T04:07:02.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:07:03.058 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:07:03.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:07:03.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:07:03.557 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:07:03.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:07:04.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:07:05.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-05-06T04:07:05.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:07:05.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:07:05.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:07:05.901 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:07:05.902 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:07:05.902 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:07:05.902 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:07:05.902 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:07:05.902 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:07:05.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:07:05.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:07:05.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:07:05.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:07:05.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:07:05.993 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:07:06.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:07:06.480 INFO:tasks.workunit.client.0.smithi163.stderr:1' 2024-05-06T04:07:06.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:07:06.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:07:06.480 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:07:06.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-05-06T04:07:06.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-05-06T04:07:06.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-05-06T04:07:06.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:07:06.595 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:07:06.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-05-06T04:07:06.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-05-06T04:07:06.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493 1-42949672969' 2024-05-06T04:07:06.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:07:06.710 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-05-06T04:07:06.710 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:07:06.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:07:06.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-05-06T04:07:06.711 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:07:06.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-05-06T04:07:06.712 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836493 2024-05-06T04:07:06.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-05-06T04:07:06.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:07:07.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836493 2024-05-06T04:07:07.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:07:08.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:07:08.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:07:08.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836493 2024-05-06T04:07:08.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:07:08.574 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-05-06T04:07:08.574 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:07:08.575 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:07:08.575 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-05-06T04:07:08.575 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:07:08.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-05-06T04:07:08.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-05-06T04:07:08.577 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672969 2024-05-06T04:07:08.577 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:07:09.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-05-06T04:07:09.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:07:09.076 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:07:09.076 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:07:09.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:07:10.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:07:10.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:07:10.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:07:10.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:07:10.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:07:10.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:07:10.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:07:10.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-05-06T04:07:10.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:07:11.069 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:07:11.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:07:11.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:07:12.025 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:07:12.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:07:13.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:07:14.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-05-06T04:07:14.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-05-06T04:07:14.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-05-06T04:07:14.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-05-06T04:07:14.215 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-05-06T04:07:14.215 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:0' 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:07:14.660 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:07:14.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:07:14.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:07:14.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:07:14.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:07:14.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:07:14.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:07:14.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:07:14.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:07:14.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:07:14.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:07:14.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:14.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:07:14.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:07:14.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:14.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:07:14.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:07:14.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:07:14.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:07:14.780 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:14.781 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:14.781 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_pg td/ceph-helpers 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:07:14.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:07:14.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:07:14.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:07:14.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:07:14.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:07:14.785 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:07:14.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:07:14.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:07:14.786 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:07:14.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:14.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:07:14.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:07:14.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:14.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:07:14.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:07:14.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:07:14.789 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:07:14.789 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:14.790 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:14.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:07:14.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:07:14.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:07:14.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:07:14.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:07:14.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:14.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:14.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:07:14.793 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:07:14.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:07:15.253 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:15.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:07:15.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:07:15.455 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:07:15.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:15.536 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:15.537 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:07:15.537 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:07:15.537 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:07:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:07:16.459 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:07:16.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:07:16.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:16.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3e774dc-85be-4edd-b5b2-b33e575151ed 2024-05-06T04:07:16.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b3e774dc-85be-4edd-b5b2-b33e575151ed' 2024-05-06T04:07:16.494 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 b3e774dc-85be-4edd-b5b2-b33e575151ed 2024-05-06T04:07:16.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:07:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0VzhmuGw0HhAAuwfsJoemWp094wn2sEUtYw== 2024-05-06T04:07:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0VzhmuGw0HhAAuwfsJoemWp094wn2sEUtYw=="}' 2024-05-06T04:07:16.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3e774dc-85be-4edd-b5b2-b33e575151ed -i td/ceph-helpers/0/new.json 2024-05-06T04:07:16.787 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:07:16.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:07:16.798 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0VzhmuGw0HhAAuwfsJoemWp094wn2sEUtYw== --osd-uuid b3e774dc-85be-4edd-b5b2-b33e575151ed 2024-05-06T04:07:16.827 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.824+0000 7f35c6412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:16.855 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.853+0000 7f35c6412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:16.856 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.854+0000 7f35c6412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:16.857 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.854+0000 7f35c6412640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:07:16.857 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.854+0000 7f35c6412640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:07:16.857 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.854+0000 7f35c6412640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:07:16.887 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:16.884+0000 7f35c6412640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:07:22.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:07:22.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:07:22.409 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:07:22.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:07:22.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:07:22.981 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:07:22.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:07:22.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:07:22.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:07:22.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:07:22.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:07:23.007 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:23.005+0000 7f0323c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:23.009 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:23.006+0000 7f0323c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:23.010 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:23.008+0000 7f0323c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:23.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:07:23.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:07:23.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:07:23.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:07:23.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:07:23.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:23.378 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:07:23.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:07:23.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:23.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:23.524 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:23.522+0000 7f0323c19640 -1 Falling back to public interface 2024-05-06T04:07:23.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:24.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:24.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:24.773 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:07:24.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:07:24.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:24.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:25.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:25.896 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:25.894+0000 7f0323c19640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:07:26.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:26.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:26.201 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:07:26.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:07:26.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:26.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:26.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:27.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:27.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:27.666 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:07:27.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:07:27.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:27.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:28.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:29.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:29.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:29.074 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:07:29.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:07:29.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:29.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:29.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:30.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:30.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:30.484 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:07:30.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:07:30.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:30.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:30.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:31.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:31.961 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:07:31.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:31.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:07:31.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:31.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:32.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:33.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:33.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:33.380 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:07:33.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:07:33.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:33.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:33.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:34.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:34.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:34.831 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:07:34.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:07:34.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:34.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:35.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:36.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:36.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:36.248 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:07:36.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:07:36.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:36.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:36.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:37.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:37.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:37.768 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:07:37.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:07:37.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:37.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:38.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:39.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:39.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:39.569 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:07:39.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:07:39.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:39.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:40.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:41.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:41.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:41.225 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:07:41.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:07:41.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:41.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:41.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:43.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:43.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:43.001 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:07:43.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:07:43.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:43.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:44.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:45.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:45.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:45.011 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:07:45.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:07:45.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:45.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:46.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:47.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:47.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:47.071 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:07:47.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:07:47.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:47.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:48.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:07:49.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:07:49.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:07:49.059 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:07:49.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:07:49.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:07:49.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:07:49.521 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1490196565,v1:127.0.0.1:6803/1490196565] [v2:127.0.0.1:6804/1490196565,v1:127.0.0.1:6805/1490196565] exists,up b3e774dc-85be-4edd-b5b2-b33e575151ed 2024-05-06T04:07:49.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:07:49.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:07:49.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:07:49.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-05-06T04:07:49.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:07:49.944 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:07:49.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:07:49.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:07:50.456 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:07:50.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:07:51.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:07:52.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-05-06T04:07:52.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:07:52.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:07:52.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:07:52.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:07:52.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:07:52.366 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:07:52.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:07:52.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:07:52.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:07:52.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:07:52.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:07:52.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:07:52.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:07:52.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:07:52.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:07:52.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:07:52.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:07:52.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:07:52.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:07:53.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836483 2024-05-06T04:07:53.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836483 2024-05-06T04:07:53.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836483' 2024-05-06T04:07:53.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:07:53.034 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836483 2024-05-06T04:07:53.034 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:07:53.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:07:53.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836483 2024-05-06T04:07:53.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:07:53.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836483 2024-05-06T04:07:53.036 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836483 2024-05-06T04:07:53.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-05-06T04:07:53.036 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:07:53.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836481 -lt 21474836483 2024-05-06T04:07:53.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:07:54.477 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:07:54.478 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:07:54.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836483 2024-05-06T04:07:54.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:07:54.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:07:54.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:07:55.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:07:55.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:07:55.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:07:55.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:07:55.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:07:55.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:07:55.491 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:07:55.491 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:07:55.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:07:55.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:07:55.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:07:55.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:07:56.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:07:56.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:07:56.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:07:56.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-05-06T04:07:56.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-05-06T04:07:56.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-05-06T04:07:56.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-05-06T04:07:56.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-05-06T04:07:56.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-05-06T04:07:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-05-06T04:07:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:07:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:07:56.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:07:56.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:07:56.903 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:07:56.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:07:56.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:07:56.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:07:57.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:07:57.008 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:07:57.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:07:57.009 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:07:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:07:57.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:07:57.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:07:57.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:57.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:07:57.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:07:57.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:57.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:07:57.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:07:57.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:07:57.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:07:57.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:57.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:57.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:07:57.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:07:57.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_primary td/ceph-helpers 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:07:57.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:07:57.022 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:07:57.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:07:57.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:07:57.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:07:57.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:07:57.023 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:07:57.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:07:57.023 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:07:57.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:07:57.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:07:57.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:07:57.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:57.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:07:57.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:07:57.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:07:57.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:07:57.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:07:57.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:07:57.027 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:07:57.028 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:57.028 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:57.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:07:57.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:07:57.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:07:57.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:07:57.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:07:57.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:57.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:57.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:07:57.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:07:57.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:07:57.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:07:57.487 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:07:57.487 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:07:57.488 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:07:57.488 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:07:57.488 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:57.488 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:57.488 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:57.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:07:57.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:07:57.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:07:57.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:07:57.690 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:07:57.690 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:07:57.690 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:07:57.690 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:07:57.690 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:07:57.691 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:07:57.691 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:07:57.691 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:57.691 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:57.691 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:07:57.692 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:07:57.692 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:57.769 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:07:57.770 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:07:57.770 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:07:57.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:07:57.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:07:58.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:07:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:07:58.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:07:58.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:07:58.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:07:58.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:07:58.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b5282ff-9c3b-4e35-8329-d273f3302c01 2024-05-06T04:07:58.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8b5282ff-9c3b-4e35-8329-d273f3302c01' 2024-05-06T04:07:58.727 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 8b5282ff-9c3b-4e35-8329-d273f3302c01 2024-05-06T04:07:58.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:07:58.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeVzhmbWwbLBAA/kKJyhKntwJ5RcCRG4Ij1w== 2024-05-06T04:07:58.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeVzhmbWwbLBAA/kKJyhKntwJ5RcCRG4Ij1w=="}' 2024-05-06T04:07:58.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b5282ff-9c3b-4e35-8329-d273f3302c01 -i td/ceph-helpers/0/new.json 2024-05-06T04:07:59.027 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:07:59.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:07:59.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeVzhmbWwbLBAA/kKJyhKntwJ5RcCRG4Ij1w== --osd-uuid 8b5282ff-9c3b-4e35-8329-d273f3302c01 2024-05-06T04:07:59.066 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.064+0000 7f70eac16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:59.087 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.085+0000 7f70eac16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:59.088 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.086+0000 7f70eac16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:07:59.089 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.086+0000 7f70eac16640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:07:59.089 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.086+0000 7f70eac16640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:07:59.089 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.086+0000 7f70eac16640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:07:59.128 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:07:59.125+0000 7f70eac16640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:08:04.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:08:04.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:08:04.764 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:08:04.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:08:04.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:08:05.313 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:08:05.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:08:05.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:08:05.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:08:05.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:08:05.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:08:05.339 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:05.337+0000 7f75c0d06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:05.376 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:05.373+0000 7f75c0d06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:05.377 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:05.375+0000 7f75c0d06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:05.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:08:05.757 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:08:05.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:05.758 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:05.929 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:05.927+0000 7f75c0d06640 -1 Falling back to public interface 2024-05-06T04:08:06.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:07.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:07.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:07.183 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:08:07.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:08:07.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:07.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:07.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:08.253 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:08.250+0000 7f75c0d06640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:08:08.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:08.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:08.618 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:08:08.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:08:08.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:08.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:09.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:10.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:10.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:10.023 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:08:10.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:08:10.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:10.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:10.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:11.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:11.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:11.452 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:08:11.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:08:11.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:11.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:11.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:12.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:12.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:12.864 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:08:12.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:08:12.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:12.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:13.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:14.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:14.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:14.392 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:08:14.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:08:14.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:14.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:14.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:15.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:15.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:15.849 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:08:15.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:08:15.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:15.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:16.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:17.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:17.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:17.374 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:08:17.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:08:17.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:17.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:17.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:18.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:18.779 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:08:18.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:18.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:08:18.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:18.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:19.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:20.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:20.210 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:08:20.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:20.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:08:20.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:20.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:20.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:21.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:21.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:21.865 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:08:21.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:08:21.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:21.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:22.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:23.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:23.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:23.522 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:08:23.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:08:23.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:23.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:24.234 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:25.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:25.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:25.236 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:08:25.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:08:25.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:25.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:26.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:27.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:27.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:27.024 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:08:27.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:08:27.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:27.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:27.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:28.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:28.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:28.494 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:08:28.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:08:28.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:28.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:29.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:30.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:30.764 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:30.764 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:08:30.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:08:30.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:30.765 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:31.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:32.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:32.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:32.408 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:08:32.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:08:32.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:32.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:32.925 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2999076439,v1:127.0.0.1:6803/2999076439] [v2:127.0.0.1:6804/2999076439,v1:127.0.0.1:6805/2999076439] exists,up 8b5282ff-9c3b-4e35-8329-d273f3302c01 2024-05-06T04:08:32.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:08:32.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:08:32.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:08:32.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-05-06T04:08:32.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:08:33.366 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:08:33.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:08:33.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:08:33.877 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:08:33.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:08:34.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:08:35.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-05-06T04:08:35.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:08:35.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:08:35.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:08:35.863 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:08:35.864 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:35.864 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:35.864 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:35.864 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:35.864 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:08:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:08:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:08:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:08:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:08:35.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:08:36.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:08:36.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:08:36.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:08:36.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:08:36.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-05-06T04:08:36.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-05-06T04:08:36.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-05-06T04:08:36.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:08:36.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-05-06T04:08:36.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:08:36.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:08:36.497 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-05-06T04:08:36.497 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:08:36.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-05-06T04:08:36.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-05-06T04:08:36.498 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836487 2024-05-06T04:08:36.498 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:08:36.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-05-06T04:08:36.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:08:37.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:08:37.887 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:08:38.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-05-06T04:08:38.289 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:08:38.289 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:08:38.289 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:08:38.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:08:38.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:08:39.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:08:39.239 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:08:39.240 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:08:39.240 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-05-06T04:08:39.782 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-06T04:08:40.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-05-06T04:08:40.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-05-06T04:08:40.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:08:40.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:08:40.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:08:40.344 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:08:40.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:08:40.345 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:08:40.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:08:40.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:08:40.346 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:08:40.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:08:40.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:08:40.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:08:40.347 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:08:40.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:08:40.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:08:40.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:08:40.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:08:40.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:40.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:40.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:08:40.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:08:40.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:08:40.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:08:40.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_get_timeout_delays td/ceph-helpers 2024-05-06T04:08:40.355 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-05-06T04:08:40.356 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.356 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.356 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.356 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-05-06T04:08:40.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-05-06T04:08:40.367 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.367 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-05-06T04:08:40.384 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-05-06T04:08:40.385 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.385 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-05-06T04:08:40.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-05-06T04:08:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.403 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.403 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-05-06T04:08:40.425 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-05-06T04:08:40.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-05-06T04:08:40.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-05-06T04:08:40.451 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.451 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.451 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.451 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.451 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-05-06T04:08:40.474 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-05-06T04:08:40.475 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.475 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.475 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.475 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.475 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-05-06T04:08:40.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-05-06T04:08:40.504 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.504 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.505 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-05-06T04:08:40.541 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-05-06T04:08:40.541 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.542 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-05-06T04:08:40.578 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-05-06T04:08:40.579 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.579 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-05-06T04:08:40.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-05-06T04:08:40.666 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.666 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.666 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.666 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.666 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-05-06T04:08:40.740 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-05-06T04:08:40.740 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:08:40.741 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:08:40.741 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:08:40.741 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:08:40.741 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_is_clean td/ceph-helpers 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:08:40.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:08:40.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:08:40.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:08:40.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:08:40.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:08:40.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:08:40.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:08:40.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:08:40.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:08:40.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:08:40.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:08:40.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:08:40.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:08:40.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:08:40.971 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:08:40.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:08:40.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:08:40.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:08:40.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:08:40.973 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:40.974 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:40.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:08:40.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:08:40.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:08:40.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:08:40.975 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:08:40.975 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:40.975 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:40.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:08:40.977 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:08:40.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:08:40.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:08:40.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:08:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:08:41.442 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:08:41.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:08:41.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:08:41.625 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:08:41.625 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:08:41.625 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:08:41.626 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:08:41.626 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:08:41.626 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:08:41.626 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:08:41.626 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:08:41.627 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:08:41.627 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:41.627 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:41.627 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:08:41.627 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:08:41.627 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:08:41.705 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:08:41.705 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:08:41.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:08:41.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-05-06T04:08:41.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:08:41.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:08:41.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:08:41.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:08:41.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:08:41.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:42.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:08:42.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:08:42.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:08:42.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:08:42.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:08:42.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:08:42.672 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:08:42.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=092ee1ce-d1e0-4638-a382-7d3414511b86 2024-05-06T04:08:42.673 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 092ee1ce-d1e0-4638-a382-7d3414511b86 2024-05-06T04:08:42.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 092ee1ce-d1e0-4638-a382-7d3414511b86' 2024-05-06T04:08:42.674 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:08:42.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKVzhm55/hKBAAvB8rEOzBbdLRE86JSuCdbA== 2024-05-06T04:08:42.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKVzhm55/hKBAAvB8rEOzBbdLRE86JSuCdbA=="}' 2024-05-06T04:08:42.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 092ee1ce-d1e0-4638-a382-7d3414511b86 -i td/ceph-helpers/0/new.json 2024-05-06T04:08:42.958 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:08:42.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:08:42.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKVzhm55/hKBAAvB8rEOzBbdLRE86JSuCdbA== --osd-uuid 092ee1ce-d1e0-4638-a382-7d3414511b86 2024-05-06T04:08:42.997 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:42.995+0000 7f945b011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:43.026 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:43.023+0000 7f945b011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:43.027 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:43.024+0000 7f945b011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:43.027 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:43.025+0000 7f945b011640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:08:43.027 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:43.025+0000 7f945b011640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:08:43.027 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:43.025+0000 7f945b011640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:08:43.057 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:43.055+0000 7f945b011640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:08:48.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:08:48.555 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:08:48.555 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:08:48.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:08:48.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:08:49.134 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:08:49.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:08:49.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:08:49.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:08:49.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:08:49.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:08:49.160 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:49.158+0000 7fafbaab1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:49.162 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:49.160+0000 7fafbaab1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:49.163 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:49.161+0000 7fafbaab1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:08:49.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:08:49.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:08:49.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:08:49.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:08:49.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:08:49.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:49.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:08:49.532 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:08:49.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:49.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:49.684 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:49.682+0000 7fafbaab1640 -1 Falling back to public interface 2024-05-06T04:08:49.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:50.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:50.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:50.963 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:08:50.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:08:50.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:50.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:51.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:52.016 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:08:52.014+0000 7fafbaab1640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:08:52.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:52.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:52.397 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:08:52.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:08:52.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:52.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:52.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:53.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:53.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:53.794 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:08:53.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:08:53.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:53.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:54.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:55.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:55.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:55.225 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:08:55.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:08:55.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:55.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:55.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:56.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:56.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:56.632 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:08:56.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:08:56.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:56.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:57.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:58.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:58.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:58.103 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:08:58.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:08:58.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:58.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:58.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:08:59.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:08:59.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:08:59.525 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:08:59.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:08:59.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:08:59.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:08:59.939 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:00.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:00.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:00.941 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:09:00.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:09:00.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:00.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:01.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:02.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:02.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:02.369 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:09:02.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:09:02.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:02.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:02.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:03.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:03.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:03.801 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:09:03.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:09:03.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:03.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:04.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:05.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:05.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:05.502 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:09:05.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:09:05.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:05.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:06.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:07.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:07.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:07.097 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:09:07.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:09:07.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:07.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:08.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:09.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:09.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:09.127 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:09:09.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:09:09.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:09.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:09.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:10.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:10.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:10.996 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:09:10.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:09:10.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:10.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:11.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:12.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:12.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:12.865 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:09:12.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:09:12.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:12.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:13.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:14.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:14.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:14.308 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:09:14.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:09:14.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:14.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:14.813 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4123162753,v1:127.0.0.1:6803/4123162753] [v2:127.0.0.1:6804/4123162753,v1:127.0.0.1:6805/4123162753] exists,up 092ee1ce-d1e0-4638-a382-7d3414511b86 2024-05-06T04:09:14.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:09:14.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:09:14.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:09:14.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-05-06T04:09:14.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:09:15.341 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:09:15.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:09:15.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:09:15.854 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:09:15.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:09:16.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:09:18.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-05-06T04:09:18.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:09:18.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:09:18.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:09:18.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:09:18.040 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:09:18.040 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:09:18.040 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:09:18.040 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:09:18.040 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:09:18.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:09:18.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:09:18.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:09:18.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:09:18.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:09:18.132 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:09:18.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:09:18.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:09:18.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:09:18.522 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:09:18.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836492 2024-05-06T04:09:18.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836492 2024-05-06T04:09:18.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836492' 2024-05-06T04:09:18.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:09:18.638 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836492 2024-05-06T04:09:18.638 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:09:18.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:09:18.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836492 2024-05-06T04:09:18.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:09:18.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836492 2024-05-06T04:09:18.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-05-06T04:09:18.641 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836492 2024-05-06T04:09:18.641 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:09:19.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836492 2024-05-06T04:09:19.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:09:19.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:09:19.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:09:19.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:09:19.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:09:19.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:09:19.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:09:19.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:09:19.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:09:19.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:09:19.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:09:20.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:09:20.042 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:09:20.042 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:09:20.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:09:20.576 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:09:21.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:09:21.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-05-06T04:09:21.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:09:21.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:09:21.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:09:21.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:09:21.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:09:21.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:09:21.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:09:21.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:09:21.632 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:09:21.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:09:21.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:09:21.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:09:21.633 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:09:21.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:09:21.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:09:21.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:09:21.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:09:21.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:21.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:21.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:09:21.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:09:21.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:09:21.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_kill_daemon td/ceph-helpers 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:09:21.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:09:21.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:09:21.644 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:09:21.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:09:21.644 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:09:21.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:09:21.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:09:21.645 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:09:21.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:09:21.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:09:21.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:09:21.647 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:09:21.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:09:21.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:09:21.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:09:21.649 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:09:21.649 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:21.649 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:21.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:09:21.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:09:21.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:09:21.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:09:21.651 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:09:21.651 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:21.651 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:21.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:09:21.653 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:09:21.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:09:22.133 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:09:22.133 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:09:22.133 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:09:22.133 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:09:22.133 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:22.134 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:22.134 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:09:22.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:09:22.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:09:22.319 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:09:22.320 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:22.320 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:22.320 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:09:22.320 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:09:22.320 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:09:22.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:09:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:09:22.403 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:09:22.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-05-06T04:09:22.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:09:22.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:09:22.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:09:22.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:09:22.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:09:22.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:09:23.331 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:09:23.331 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:09:23.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:09:23.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-05-06T04:09:23.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:09:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:09:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:09:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:09:23.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:09:23.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c3a6623-c61a-47c3-a3d5-42d49c5d1f2b 2024-05-06T04:09:23.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5c3a6623-c61a-47c3-a3d5-42d49c5d1f2b' 2024-05-06T04:09:23.366 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 5c3a6623-c61a-47c3-a3d5-42d49c5d1f2b 2024-05-06T04:09:23.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:09:23.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzVzhmKV+FFhAA8gi8yq1vymdaAc0+VJlOYg== 2024-05-06T04:09:23.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzVzhmKV+FFhAA8gi8yq1vymdaAc0+VJlOYg=="}' 2024-05-06T04:09:23.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c3a6623-c61a-47c3-a3d5-42d49c5d1f2b -i td/ceph-helpers/0/new.json 2024-05-06T04:09:23.675 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:09:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:09:23.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzVzhmKV+FFhAA8gi8yq1vymdaAc0+VJlOYg== --osd-uuid 5c3a6623-c61a-47c3-a3d5-42d49c5d1f2b 2024-05-06T04:09:23.714 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.711+0000 7f66cc30a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:09:23.743 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.741+0000 7f66cc30a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:09:23.745 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.742+0000 7f66cc30a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:09:23.745 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.743+0000 7f66cc30a640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:09:23.745 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.743+0000 7f66cc30a640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:09:23.745 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.743+0000 7f66cc30a640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:09:23.783 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:23.781+0000 7f66cc30a640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:09:29.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:09:29.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:09:29.339 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:09:29.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:09:29.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:09:29.908 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:09:29.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:09:29.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:09:29.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:09:29.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:09:29.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:09:29.934 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:29.931+0000 7f116e633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:09:29.936 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:29.933+0000 7f116e633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:09:29.937 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:29.934+0000 7f116e633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:09:30.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:09:30.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:09:30.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:09:30.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:09:30.303 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:09:30.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:09:30.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:30.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:09:30.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:30.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:30.451 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:30.449+0000 7f116e633640 -1 Falling back to public interface 2024-05-06T04:09:30.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:31.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:31.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:31.697 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:09:31.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:09:31.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:31.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:32.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:32.789 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:09:32.786+0000 7f116e633640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:09:33.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:33.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:33.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:09:33.130 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:09:33.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:33.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:33.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:34.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:34.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:34.546 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:09:34.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:09:34.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:34.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:34.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:35.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:35.956 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:09:35.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:09:35.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:35.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:36.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:37.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:37.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:37.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:09:37.366 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:09:37.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:37.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:37.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:38.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:38.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:38.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:09:38.833 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:09:38.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:38.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:39.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:40.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:40.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:40.240 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:09:40.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:09:40.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:40.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:40.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:41.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:41.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:41.716 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:09:41.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:09:41.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:41.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:42.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:43.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:43.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:43.148 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:09:43.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:09:43.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:43.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:43.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:44.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:44.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:44.589 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:09:44.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:09:44.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:44.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:45.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:46.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:46.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:46.316 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:09:46.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:09:46.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:46.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:46.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:47.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:47.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:47.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:09:47.999 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:09:47.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:47.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:48.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:49.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:49.833 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:49.834 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:09:49.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:09:49.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:49.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:50.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:51.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:51.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:51.499 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:09:51.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:09:51.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:51.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:52.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:53.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:53.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:53.019 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:09:53.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:09:53.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:53.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:54.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:09:55.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:09:55.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:09:55.164 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:09:55.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:09:55.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:09:55.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:09:55.566 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3655376752,v1:127.0.0.1:6803/3655376752] [v2:127.0.0.1:6804/3655376752,v1:127.0.0.1:6805/3655376752] exists,up 5c3a6623-c61a-47c3-a3d5-42d49c5d1f2b 2024-05-06T04:09:55.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:09:55.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:09:55.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:09:55.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-05-06T04:09:55.568 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-05-06T04:09:55.568 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-05-06T04:09:55.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-05-06T04:09:55.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-05-06T04:09:55.568 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=121510 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 121510 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-05-06T04:09:55.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-05-06T04:09:55.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-05-06T04:09:56.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 121510 2024-05-06T04:09:56.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-05-06T04:09:56.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-05-06T04:09:56.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-05-06T04:09:56.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-05-06T04:09:56.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-05-06T04:09:56.572 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=121510 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 121510 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-05-06T04:09:56.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-05-06T04:09:56.575 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-05-06T04:09:56.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 121510 2024-05-06T04:09:56.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-05-06T04:09:56.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-05-06T04:09:56.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-05-06T04:09:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 121510 2024-05-06T04:09:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-05-06T04:09:56.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-05-06T04:09:56.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-05-06T04:09:56.878 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-05-06T04:09:56.878 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-05-06T04:09:56.878 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=120563 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 120563 2024-05-06T04:09:56.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-05-06T04:09:56.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-05-06T04:09:56.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-05-06T04:09:56.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 120563 2024-05-06T04:09:56.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-05-06T04:09:56.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-05-06T04:09:56.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-05-06T04:09:56.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-05-06T04:09:56.983 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-05-06T04:09:56.983 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-05-06T04:09:56.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-05-06T04:09:56.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-05-06T04:09:56.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=120478 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 120478 2024-05-06T04:09:56.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-05-06T04:09:56.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-05-06T04:09:56.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-05-06T04:09:57.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-05-06T04:09:57.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 120478 2024-05-06T04:09:57.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-05-06T04:09:57.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-05-06T04:09:57.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-05-06T04:09:57.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-05-06T04:10:02.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:02.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:02.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:02.097 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:10:02.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:10:02.098 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:10:02.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:10:02.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:10:02.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:10:02.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:02.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:10:02.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:10:02.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:02.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:10:02.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:10:02.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:10:02.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:10:02.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:02.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:02.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:10:02.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_kill_daemons td/ceph-helpers 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:02.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:02.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:02.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:02.113 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:10:02.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:10:02.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:10:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:10:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:10:02.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:10:02.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:02.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:10:02.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:10:02.117 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:02.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:10:02.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:10:02.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:10:02.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:10:02.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:02.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:02.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:10:02.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:10:02.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:10:02.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:10:02.122 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:10:02.122 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:02.123 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:02.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:10:02.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:10:02.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:10:02.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:10:02.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:10:02.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:10:02.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:10:02.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:10:02.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:10:02.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:10:02.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:10:02.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:10:02.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:10:02.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:10:02.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:10:02.603 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:10:02.603 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:02.603 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:02.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:02.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:10:02.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:10:02.804 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:10:02.804 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:10:02.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:10:02.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:10:02.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:10:02.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:10:02.898 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:10:02.899 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:10:02.900 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:10:02.900 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:02.900 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:02.900 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:10:02.900 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:10:02.900 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:10:02.982 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:03.834 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:03.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:10:03.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:10:03.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-05-06T04:10:03.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:10:03.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:10:03.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:10:03.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:10:03.866 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:10:03.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64785127-d20b-4312-b613-d2eaaf56b360 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 64785127-d20b-4312-b613-d2eaaf56b360' 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 64785127-d20b-4312-b613-d2eaaf56b360 2024-05-06T04:10:03.868 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:10:03.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbWDhmXb99NBAASSnp5871UOTf6EXvNqlqPw== 2024-05-06T04:10:03.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbWDhmXb99NBAASSnp5871UOTf6EXvNqlqPw=="}' 2024-05-06T04:10:03.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64785127-d20b-4312-b613-d2eaaf56b360 -i td/ceph-helpers/0/new.json 2024-05-06T04:10:04.169 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:10:04.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:10:04.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbWDhmXb99NBAASSnp5871UOTf6EXvNqlqPw== --osd-uuid 64785127-d20b-4312-b613-d2eaaf56b360 2024-05-06T04:10:04.208 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.205+0000 7f8780b85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:04.238 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.235+0000 7f8780b85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:04.239 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.236+0000 7f8780b85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:04.239 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.236+0000 7f8780b85640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:10:04.240 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.236+0000 7f8780b85640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:10:04.240 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.236+0000 7f8780b85640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:10:04.277 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:04.275+0000 7f8780b85640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:10:09.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:10:09.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:10:09.914 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:10:09.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:10:09.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:10:10.470 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:10:10.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:10:10.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:10:10.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:10:10.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:10:10.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:10:10.497 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:10.493+0000 7f8531ca8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:10.503 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:10.499+0000 7f8531ca8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:10.504 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:10.501+0000 7f8531ca8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:10:10.909 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:10:10.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:10.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:11.046 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:11.042+0000 7f8531ca8640 -1 Falling back to public interface 2024-05-06T04:10:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:12.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:12.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:12.345 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:10:12.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:10:12.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:12.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:13.389 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:13.386+0000 7f8531ca8640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:10:13.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:13.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:13.789 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:10:13.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:10:13.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:13.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:14.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:15.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:15.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:15.412 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:10:15.412 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:10:15.413 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:15.413 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:15.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:16.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:16.820 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:10:16.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:16.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:10:16.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:16.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:17.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:18.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:18.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:18.256 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:10:18.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:10:18.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:18.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:18.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:19.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:19.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:19.674 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:10:19.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:10:19.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:19.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:20.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:21.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:21.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:21.112 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:10:21.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:10:21.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:21.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:21.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:22.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:22.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:22.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:10:22.528 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:10:22.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:22.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:22.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:23.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:23.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:23.944 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:10:23.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:10:23.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:23.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:24.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:25.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:25.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:25.432 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:10:25.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:10:25.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:25.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:26.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:27.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:27.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:27.318 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:10:27.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:10:27.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:27.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:28.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:29.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:29.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:29.080 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:10:29.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:10:29.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:29.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:29.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:30.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:30.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:30.883 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:10:30.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:10:30.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:30.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:31.559 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:32.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:32.561 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:10:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:10:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:32.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:33.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:34.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:34.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:34.288 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:10:34.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:10:34.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:34.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:35.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:36.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:36.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:36.351 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:10:36.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:10:36.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:36.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:36.749 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3191068512,v1:127.0.0.1:6803/3191068512] [v2:127.0.0.1:6804/3191068512,v1:127.0.0.1:6805/3191068512] exists,up 64785127-d20b-4312-b613-d2eaaf56b360 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:36.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:36.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:36.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:37.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-05-06T04:10:37.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-05-06T04:10:37.756 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:37.756 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:37.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:37.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:37.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:38.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:38.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-05-06T04:10:38.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:38.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:38.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:38.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:38.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:38.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:38.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-05-06T04:10:38.171 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:38.171 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:38.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:38.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:38.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:38.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:38.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-05-06T04:10:43.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-05-06T04:10:43.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:10:43.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:10:43.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:10:43.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:43.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:43.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:43.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:43.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:43.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:43.286 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:10:43.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:10:43.287 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:10:43.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:10:43.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:10:43.289 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:10:43.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:43.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:10:43.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:10:43.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:43.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:10:43.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:10:43.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:10:43.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:10:43.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:43.299 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:43.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:10:43.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_objectstore_tool td/ceph-helpers 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:10:43.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:10:43.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:10:43.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:10:43.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:10:43.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:10:43.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:10:43.307 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:10:43.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:10:43.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:10:43.308 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:10:43.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:43.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:10:43.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:10:43.308 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:10:43.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:10:43.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:10:43.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:10:43.310 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:10:43.310 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:43.311 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:43.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:10:43.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:10:43.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:10:43.311 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:10:43.312 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:10:43.312 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:43.312 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:43.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:10:43.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:10:43.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:10:43.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:10:43.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:10:43.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:10:43.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:10:43.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:10:43.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:10:43.771 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:10:43.771 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:43.772 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:43.772 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:43.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:10:43.989 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:10:43.990 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:10:43.990 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:10:43.990 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:10:43.990 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:10:43.991 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:10:43.992 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:10:44.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:10:44.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:10:44.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:10:44.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:10:44.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:10:44.081 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:10:44.081 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:10:44.081 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:10:44.081 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:10:44.082 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:44.082 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:44.082 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:10:44.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:10:44.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:10:44.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-05-06T04:10:44.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-05-06T04:10:44.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:10:44.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:10:44.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:10:44.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:10:44.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:10:44.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:45.012 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:10:45.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:10:45.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:10:45.043 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:10:45.044 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:10:45.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82ff704a-27f8-4361-9937-74cdbb063cf3 2024-05-06T04:10:45.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 82ff704a-27f8-4361-9937-74cdbb063cf3' 2024-05-06T04:10:45.045 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 82ff704a-27f8-4361-9937-74cdbb063cf3 2024-05-06T04:10:45.046 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:10:45.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFWDhm/IZ/AxAAJB6NPcI84IUqhyfQTgp3rA== 2024-05-06T04:10:45.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFWDhm/IZ/AxAAJB6NPcI84IUqhyfQTgp3rA=="}' 2024-05-06T04:10:45.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82ff704a-27f8-4361-9937-74cdbb063cf3 -i td/ceph-helpers/0/new.json 2024-05-06T04:10:45.340 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:10:45.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:10:45.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFWDhm/IZ/AxAAJB6NPcI84IUqhyfQTgp3rA== --osd-uuid 82ff704a-27f8-4361-9937-74cdbb063cf3 2024-05-06T04:10:45.381 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.377+0000 7f7293043640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:45.409 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.406+0000 7f7293043640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:45.410 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.408+0000 7f7293043640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:45.411 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.408+0000 7f7293043640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:10:45.411 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.408+0000 7f7293043640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:10:45.411 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.408+0000 7f7293043640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:10:45.449 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:45.446+0000 7f7293043640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:10:51.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:10:51.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:10:51.007 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:10:51.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:10:51.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:10:51.614 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:10:51.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:10:51.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:10:51.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:10:51.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:10:51.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:10:51.640 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:51.638+0000 7faa02225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:51.642 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:51.640+0000 7faa02225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:51.643 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:51.641+0000 7faa02225640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:10:52.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:10:52.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:10:52.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:10:52.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:10:52.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:10:52.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:52.060 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:10:52.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:10:52.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:52.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:52.178 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:52.174+0000 7faa02225640 -1 Falling back to public interface 2024-05-06T04:10:52.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:53.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:53.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:53.494 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:10:53.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:10:53.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:53.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:53.935 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:54.531 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:10:54.527+0000 7faa02225640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:10:54.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:54.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:54.937 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:10:54.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:10:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:54.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:55.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:56.567 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:56.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:56.568 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:10:56.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:10:56.568 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:56.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:56.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:57.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:57.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:57.999 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:10:58.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:10:58.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:58.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:58.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:10:59.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:10:59.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:10:59.430 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:10:59.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:10:59.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:10:59.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:10:59.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:00.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:00.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:00.858 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:11:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:11:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:01.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:02.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:02.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:02.381 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:11:02.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:11:02.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:02.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:02.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:03.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:03.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:03.816 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:11:03.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:11:03.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:03.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:04.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:05.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:05.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:05.261 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:11:05.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:11:05.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:05.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:05.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:06.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:06.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:06.666 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:11:06.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:11:06.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:06.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:07.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:08.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:08.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:08.182 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:11:08.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:11:08.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:08.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:08.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:09.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:09.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:09.719 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:11:09.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:11:09.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:09.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:10.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:11.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:11.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:11.370 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:11:11.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:11:11.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:11.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:12.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:13.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:13.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:13.173 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:11:13.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:11:13.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:13.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:13.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:14.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:14.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:14.722 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:11:14.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:11:14.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:14.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:15.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:16.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:16.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:16.830 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:11:16.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:11:16.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:16.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:17.250 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4155808745,v1:127.0.0.1:6803/4155808745] [v2:127.0.0.1:6804/4155808745,v1:127.0.0.1:6805/4155808745] exists,up 82ff704a-27f8-4361-9937-74cdbb063cf3 2024-05-06T04:11:17.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:11:17.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:11:17.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:11:17.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-05-06T04:11:17.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:11:17.725 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:11:17.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:11:17.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:11:18.238 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:11:18.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:11:19.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:11:20.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-05-06T04:11:20.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:11:20.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:11:20.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:11:20.255 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:11:20.256 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:11:20.256 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:11:20.256 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:11:20.256 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:11:20.256 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:11:20.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:11:20.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:11:20.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:11:20.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:11:20.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:11:20.347 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:11:20.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:11:20.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:11:20.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:11:20.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:11:20.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-05-06T04:11:20.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-05-06T04:11:20.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-05-06T04:11:20.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:11:20.861 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-05-06T04:11:20.861 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:11:20.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:11:20.862 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-05-06T04:11:20.862 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:11:20.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-05-06T04:11:20.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-05-06T04:11:20.863 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836487 2024-05-06T04:11:20.863 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:11:21.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836487 2024-05-06T04:11:21.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:11:21.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:11:21.291 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:11:21.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:11:21.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:11:21.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:11:21.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:11:21.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:11:21.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:11:21.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:11:21.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:11:22.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:11:22.306 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:11:22.306 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:11:22.306 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:11:22.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:11:22.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:11:22.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:11:22.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-05-06T04:11:22.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-05-06T04:11:22.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-05-06T04:11:22.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-06T04:11:22.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-05-06T04:11:22.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-05-06T04:11:22.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:11:22.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:11:23.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:11:23.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-05-06T04:11:23.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-05-06T04:11:23.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-06T04:11:23.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-05-06T04:11:23.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-06T04:11:23.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-05-06T04:11:23.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-05-06T04:11:23.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:11:23.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:11:23.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:23.683 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:11:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:start osd.0 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:11:23.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:11:23.704 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:23.701+0000 7f4371463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:23.705 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:23.703+0000 7f4371463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:23.706 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:23.704+0000 7f4371463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:24.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T04:11:24.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:11:24.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:0 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:24.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:24.223 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:24.221+0000 7f4371463640 -1 Falling back to public interface 2024-05-06T04:11:24.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:25.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:25.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:25.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:11:25.594 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T04:11:25.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:25.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:26.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:26.952 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:26.949+0000 7f4371463640 -1 osd.0 13 log_to_monitors true 2024-05-06T04:11:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:11:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:2 2024-05-06T04:11:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:27.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:27.455 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 up in weight 1 up_from 17 up_thru 7 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/2641285218,v1:127.0.0.1:6803/2641285218] [v2:127.0.0.1:6804/2641285218,v1:127.0.0.1:6805/2641285218] exists,up 82ff704a-27f8-4361-9937-74cdbb063cf3 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:11:27.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:11:27.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:11:27.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:11:27.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:11:27.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:11:27.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:11:27.534 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:11:28.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:11:28.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:11:28.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:11:28.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:11:28.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=73014444035 2024-05-06T04:11:28.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 73014444035 2024-05-06T04:11:28.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-73014444035' 2024-05-06T04:11:28.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:11:28.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-73014444035 2024-05-06T04:11:28.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:11:28.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:11:28.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-73014444035 2024-05-06T04:11:28.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:11:28.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=73014444035 2024-05-06T04:11:28.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 73014444035' 2024-05-06T04:11:28.128 INFO:tasks.workunit.client.0.smithi163.stderr:waiting osd.0 seq 73014444035 2024-05-06T04:11:28.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:11:28.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 73014444035 2024-05-06T04:11:28.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:11:29.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:11:29.554 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:11:30.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 73014444035 -lt 73014444035 2024-05-06T04:11:30.013 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:11:30.013 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:11:30.013 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:11:30.592 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:11:31.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:11:31.003 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:11:31.003 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:11:31.003 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:11:31.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-06T04:11:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:11:31.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:11:31.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:11:31.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-05-06T04:11:31.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-05-06T04:11:31.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-06T04:11:31.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-05-06T04:11:31.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-06T04:11:31.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-05-06T04:11:31.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-05-06T04:11:31.986 INFO:tasks.workunit.client.0.smithi163.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-05-06T04:11:32.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:11:32.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:11:32.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:11:32.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:11:32.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:11:32.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:11:32.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:11:32.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:11:32.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:11:32.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:11:32.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:11:32.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:11:32.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:11:32.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:11:32.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:11:32.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:11:32.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:11:32.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:32.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:32.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_pg_schedule_scrub td/ceph-helpers 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_schedule_scrub: local dir=td/ceph-helpers 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_schedule_scrub: setup td/ceph-helpers 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:11:32.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:11:32.373 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:11:32.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:11:32.374 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:11:32.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:11:32.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:11:32.375 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:11:32.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:11:32.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:11:32.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:11:32.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:11:32.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:11:32.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:11:32.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:11:32.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:11:32.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:32.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:32.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:11:32.380 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:11:32.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:11:32.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:11:32.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:11:32.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:32.382 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:32.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:11:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:11:32.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:11:32.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:11:32.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:11:32.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:32.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:11:32.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:11:33.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:11:33.049 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:11:33.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:11:33.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:11:33.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:11:33.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:11:33.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:11:33.052 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:11:33.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:11:33.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:11:33.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:11:33.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_schedule_scrub: run_mgr td/ceph-helpers x 2024-05-06T04:11:33.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:11:33.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:11:33.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:11:33.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:11:33.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:11:33.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:11:34.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:11:34.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:11:34.071 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:11:34.071 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:11:34.071 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:34.072 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:34.072 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:11:34.072 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:11:34.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:11:34.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2024-05-06T04:11:34.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:11:34.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:11:34.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:11:34.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:11:34.103 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:11:34.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:11:34.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce6fd4b8-5cb6-429d-b406-0d031e5d19cd 2024-05-06T04:11:34.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce6fd4b8-5cb6-429d-b406-0d031e5d19cd' 2024-05-06T04:11:34.106 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 ce6fd4b8-5cb6-429d-b406-0d031e5d19cd 2024-05-06T04:11:34.106 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:11:34.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB2WDhmFhQUBxAA2cQupwMejakFyoH2wUAWmA== 2024-05-06T04:11:34.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB2WDhmFhQUBxAA2cQupwMejakFyoH2wUAWmA=="}' 2024-05-06T04:11:34.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce6fd4b8-5cb6-429d-b406-0d031e5d19cd -i td/ceph-helpers/0/new.json 2024-05-06T04:11:34.397 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:11:34.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:11:34.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2WDhmFhQUBxAA2cQupwMejakFyoH2wUAWmA== --osd-uuid ce6fd4b8-5cb6-429d-b406-0d031e5d19cd 2024-05-06T04:11:34.437 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.434+0000 7f9315a10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:34.466 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.463+0000 7f9315a10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:34.467 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.465+0000 7f9315a10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:34.468 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.465+0000 7f9315a10640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:11:34.468 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.465+0000 7f9315a10640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:11:34.468 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.465+0000 7f9315a10640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:11:34.506 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:34.504+0000 7f9315a10640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:11:39.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:11:39.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:11:40.000 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:11:40.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:11:40.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:11:40.606 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:11:40.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:11:40.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:11:40.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:11:40.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:11:40.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:11:40.633 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:40.629+0000 7f093fd40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:40.634 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:40.631+0000 7f093fd40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:40.635 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:40.632+0000 7f093fd40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:11:41.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:11:41.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:11:41.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:11:41.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:11:41.054 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:11:41.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:11:41.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:41.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:11:41.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:41.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:41.163 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:41.161+0000 7f093fd40640 -1 Falling back to public interface 2024-05-06T04:11:41.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:42.481 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:42.481 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:42.482 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:11:42.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:11:42.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:42.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:42.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:43.525 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:11:43.523+0000 7f093fd40640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:11:43.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:43.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:43.924 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:11:43.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:11:43.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:43.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:44.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:45.441 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:45.441 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:45.441 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:11:45.441 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:11:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:45.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:45.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:46.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:46.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:46.846 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:11:46.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:11:46.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:46.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:47.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:48.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:48.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:48.402 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:11:48.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:11:48.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:48.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:48.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:49.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:49.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:49.809 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:11:49.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:11:49.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:49.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:50.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:51.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:51.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:51.265 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:11:51.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:11:51.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:51.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:51.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:52.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:52.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:52.688 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:11:52.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:11:52.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:52.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:53.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:54.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:54.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:54.094 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:11:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:11:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:54.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:55.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:55.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:55.630 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:11:55.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:11:55.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:55.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:56.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:57.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:57.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:57.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:11:57.400 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:11:57.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:57.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:58.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:11:59.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:11:59.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:11:59.031 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:11:59.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:11:59.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:11:59.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:11:59.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:00.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:00.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:00.779 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:12:00.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:12:00.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:00.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:01.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:02.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:02.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:02.834 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:12:02.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:12:02.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:02.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:03.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:04.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:04.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:04.927 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:12:04.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:12:04.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:04.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:05.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:06.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:06.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:06.870 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:12:06.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:12:06.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:06.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:07.269 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1585090205,v1:127.0.0.1:6803/1585090205] [v2:127.0.0.1:6804/1585090205,v1:127.0.0.1:6805/1585090205] exists,up ce6fd4b8-5cb6-429d-b406-0d031e5d19cd 2024-05-06T04:12:07.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:12:07.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:12:07.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:12:07.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1953: test_pg_schedule_scrub: create_rbd_pool 2024-05-06T04:12:07.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:12:07.691 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:12:07.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:12:07.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:12:08.186 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:12:08.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:12:09.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:12:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: test_pg_schedule_scrub: wait_for_clean 2024-05-06T04:12:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:12:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:12:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:12:10.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:12:10.129 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:12:10.129 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:12:10.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:12:10.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:12:10.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:12:10.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:12:10.220 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:12:10.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:12:10.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:12:10.221 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:12:10.221 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:12:10.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:12:10.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:12:10.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:12:10.656 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:12:10.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836484 2024-05-06T04:12:10.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836484 2024-05-06T04:12:10.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836484' 2024-05-06T04:12:10.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:12:10.772 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836484 2024-05-06T04:12:10.772 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:12:10.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:12:10.773 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836484 2024-05-06T04:12:10.773 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:12:10.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836484 2024-05-06T04:12:10.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-05-06T04:12:10.774 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836484 2024-05-06T04:12:10.775 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:12:11.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836482 -lt 21474836484 2024-05-06T04:12:11.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:12:12.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:12:12.200 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:12:12.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836484 2024-05-06T04:12:12.625 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:12:12.625 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:12:12.625 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:12:13.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:12:13.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:12:13.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:12:13.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:12:13.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:12:13.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:12:13.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:12:13.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:12:13.569 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:12:13.569 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:12:13.569 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:12:13.569 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1955: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: pg_schedule_scrub: local pgid=1.0 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: get_last_scrub_stamp 1.0 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:14.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:14.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_schedule_scrub: local last_scrub=2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:14.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_schedule_scrub: ceph pg scrub 1.0 2024-05-06T04:12:14.960 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to scrub 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_schedule_scrub: wait_for_scrub 1.0 2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:14.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:15.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:08.024039+0000 '>' 2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:15.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:12:16.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:12:16.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:12:16.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:12:16.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:16.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:16.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:16.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:16.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:08.024039+0000 '>' 2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:16.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:12:17.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:12:17.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:12:17.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:12:17.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:17.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:17.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:17.758 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:18.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:08.024039+0000 '>' 2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:18.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:12:19.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:12:19.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:12:19.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:12:19.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:19.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:19.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:19.155 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:19.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:15.641890+0000 '>' 2024-05-06T04:12:08.024039+0000 2024-05-06T04:12:19.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-05-06T04:12:19.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: test_pg_schedule_scrub: kill_daemons td/ceph-helpers KILL osd 2024-05-06T04:12:19.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:12:19.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:12:19.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:12:19.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:12:19.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_pg_schedule_scrub: TIMEOUT=1 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_pg_schedule_scrub: pg_scrub 1.0 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=1.0 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 1.0 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:19.657 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:20.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-05-06T04:12:15.641890+0000 2024-05-06T04:12:20.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 1.0 2024-05-06T04:12:20.419 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to scrub 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 1.0 2024-05-06T04:12:15.641890+0000 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:12:15.641890+0000 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:12:20.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:12:20.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:12:20.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:15.641890+0000 '>' 2024-05-06T04:12:15.641890+0000 2024-05-06T04:12:20.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1958: test_pg_schedule_scrub: teardown td/ceph-helpers 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:12:21.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:12:21.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:12:21.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:12:21.930 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:12:21.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:12:21.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:12:21.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:12:21.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:12:21.932 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:12:21.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:12:21.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:12:21.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:12:21.934 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:12:21.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:12:21.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:12:21.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:12:21.942 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:12:21.942 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:21.942 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:21.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:12:21.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:12:21.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:12:21.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:12:21.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_pg_scrub td/ceph-helpers 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_pg_scrub: local dir=td/ceph-helpers 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_pg_scrub: setup td/ceph-helpers 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:12:21.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:12:21.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:12:21.946 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:12:21.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:12:21.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:12:21.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:12:21.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:12:21.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:12:21.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:12:21.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:12:21.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:12:21.949 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:12:21.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:12:21.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:12:21.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:21.955 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:21.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:12:21.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:12:21.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:12:21.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:12:21.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:12:21.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:12:22.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:12:22.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:12:22.421 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:12:22.421 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:12:22.422 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:22.422 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:22.422 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:12:22.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:12:22.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:12:22.621 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:12:22.621 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:12:22.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:12:22.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:12:22.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:12:22.623 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:12:22.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:22.712 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:22.713 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:12:22.713 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:12:22.713 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1909: test_pg_scrub: run_mgr td/ceph-helpers x 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:12:22.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:23.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:12:23.644 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:12:23.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: test_pg_scrub: run_osd td/ceph-helpers 0 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:12:23.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:12:23.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:12:23.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7939785b-beca-4e34-84c3-31c6e949545c 2024-05-06T04:12:23.678 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 7939785b-beca-4e34-84c3-31c6e949545c 2024-05-06T04:12:23.678 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7939785b-beca-4e34-84c3-31c6e949545c' 2024-05-06T04:12:23.678 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:12:23.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnWDhmtIIqKRAA40bpEct/Xw+GMzIcWD2ObA== 2024-05-06T04:12:23.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnWDhmtIIqKRAA40bpEct/Xw+GMzIcWD2ObA=="}' 2024-05-06T04:12:23.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7939785b-beca-4e34-84c3-31c6e949545c -i td/ceph-helpers/0/new.json 2024-05-06T04:12:24.019 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:12:24.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:12:24.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnWDhmtIIqKRAA40bpEct/Xw+GMzIcWD2ObA== --osd-uuid 7939785b-beca-4e34-84c3-31c6e949545c 2024-05-06T04:12:24.058 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.055+0000 7f140cc25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:12:24.088 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.085+0000 7f140cc25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:12:24.089 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.087+0000 7f140cc25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:12:24.090 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.087+0000 7f140cc25640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:12:24.090 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.087+0000 7f140cc25640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:12:24.090 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.087+0000 7f140cc25640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:12:24.119 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:24.117+0000 7f140cc25640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:12:29.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:12:29.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:12:29.780 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:12:29.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:12:29.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:12:30.330 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:12:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:12:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:12:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:12:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:12:30.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:12:30.356 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:30.354+0000 7f62e4a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:12:30.366 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:30.364+0000 7f62e4a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:12:30.368 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:30.365+0000 7f62e4a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:12:30.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:12:30.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:12:30.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:12:30.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:12:30.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:12:30.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:30.790 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:12:30.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:12:30.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:30.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:30.904 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:30.902+0000 7f62e4a0d640 -1 Falling back to public interface 2024-05-06T04:12:31.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:32.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:32.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:32.216 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:12:32.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:12:32.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:32.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:32.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:33.259 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:12:33.257+0000 7f62e4a0d640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:12:33.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:33.659 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:33.659 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:12:33.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:12:33.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:33.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:34.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:35.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:35.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:35.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:12:35.347 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:12:35.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:35.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:35.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:36.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:36.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:36.753 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:12:36.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:12:36.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:36.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:37.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:38.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:38.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:38.208 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:12:38.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:12:38.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:38.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:38.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:39.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:39.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:39.633 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:12:39.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:12:39.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:39.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:40.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:41.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:41.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:41.184 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:12:41.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:12:41.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:41.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:41.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:42.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:42.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:42.607 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:12:42.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:12:42.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:42.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:43.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:44.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:44.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:44.082 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:12:44.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:12:44.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:44.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:44.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:45.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:45.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:45.592 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:12:45.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:12:45.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:45.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:46.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:47.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:47.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:47.018 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:12:47.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:12:47.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:47.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:47.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:48.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:48.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:48.808 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:12:48.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:12:48.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:48.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:49.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:50.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:50.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:50.725 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:12:50.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:12:50.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:50.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:51.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:52.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:52.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:52.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:12:52.503 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:12:52.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:52.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:53.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:54.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:54.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:54.200 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:12:54.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:12:54.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:54.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:55.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:12:56.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:12:56.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:12:56.101 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:12:56.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:12:56.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:12:56.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:12:56.511 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4289229374,v1:127.0.0.1:6803/4289229374] [v2:127.0.0.1:6804/4289229374,v1:127.0.0.1:6805/4289229374] exists,up 7939785b-beca-4e34-84c3-31c6e949545c 2024-05-06T04:12:56.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:12:56.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:12:56.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:12:56.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: test_pg_scrub: create_rbd_pool 2024-05-06T04:12:56.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:12:56.968 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:12:56.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:12:56.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:12:57.662 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:12:57.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:12:58.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_pg_scrub: wait_for_clean 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:12:59.766 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:12:59.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:12:59.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:12:59.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:12:59.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:12:59.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:12:59.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:12:59.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:12:59.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:13:00.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:13:00.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:13:00.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:13:00.293 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:13:00.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-05-06T04:13:00.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-05-06T04:13:00.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-05-06T04:13:00.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:13:00.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-05-06T04:13:00.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:13:00.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:13:00.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-05-06T04:13:00.409 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:13:00.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-05-06T04:13:00.410 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836493 2024-05-06T04:13:00.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-05-06T04:13:00.411 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:13:00.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836493 2024-05-06T04:13:00.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:13:01.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:13:01.845 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:13:02.278 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836493 2024-05-06T04:13:02.279 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:13:02.279 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:13:02.279 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:13:02.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:13:02.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:13:02.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:13:02.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:13:02.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:13:02.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:13:02.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:13:02.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:13:03.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:13:03.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:13:03.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:13:03.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:13:03.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:13:03.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_pg_scrub: pg_scrub 1.0 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=1.0 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 1.0 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:03.784 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:04.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-05-06T04:12:57.302952+0000 2024-05-06T04:13:04.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 1.0 2024-05-06T04:13:04.554 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to scrub 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 1.0 2024-05-06T04:12:57.302952+0000 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:12:57.302952+0000 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:04.566 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:04.567 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:04.567 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:04.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:57.302952+0000 '>' 2024-05-06T04:12:57.302952+0000 2024-05-06T04:13:04.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:05.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:05.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:05.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:05.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:05.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:05.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:05.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:06.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:12:57.302952+0000 '>' 2024-05-06T04:12:57.302952+0000 2024-05-06T04:13:06.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:07.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:07.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:07.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:07.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:07.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:07.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:07.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:07.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:05.397608+0000 '>' 2024-05-06T04:12:57.302952+0000 2024-05-06T04:13:07.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-05-06T04:13:07.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1914: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-05-06T04:13:07.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:13:07.747 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:13:07.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:13:07.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:13:07.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_pg_scrub: TIMEOUT=1 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_pg_scrub: pg_scrub 1.0 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_scrub: local pgid=1.0 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: get_last_scrub_stamp 1.0 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:07.852 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:08.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_scrub: local last_scrub=2024-05-06T04:13:05.397608+0000 2024-05-06T04:13:08.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_scrub: ceph pg scrub 1.0 2024-05-06T04:13:08.612 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to scrub 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: pg_scrub: wait_for_scrub 1.0 2024-05-06T04:13:05.397608+0000 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:13:05.397608+0000 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:08.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:09.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:05.397608+0000 '>' 2024-05-06T04:13:05.397608+0000 2024-05-06T04:13:09.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:10.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:10.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:13:10.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-05-06T04:13:10.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1916: test_pg_scrub: teardown td/ceph-helpers 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:13:10.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:13:10.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:13:10.124 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:13:10.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:13:10.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:13:10.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:13:10.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:13:10.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:13:10.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:13:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:13:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:13:10.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:13:10.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:13:10.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:13:10.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:13:10.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:13:10.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:10.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:10.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:13:10.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:13:10.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_repair td/ceph-helpers 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_repair: local dir=td/ceph-helpers 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_repair: setup td/ceph-helpers 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:13:10.139 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:13:10.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:13:10.140 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:13:10.141 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:13:10.141 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:13:10.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:13:10.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:13:10.142 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:13:10.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:13:10.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:13:10.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:13:10.144 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:13:10.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:13:10.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:13:10.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:13:10.147 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:13:10.147 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:10.147 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:10.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:13:10.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:13:10.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:13:10.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:13:10.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:13:10.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:10.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:10.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:13:10.154 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:13:10.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:13:10.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:13:10.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:13:10.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:13:10.615 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:13:10.615 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:10.615 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:10.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:13:10.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:13:10.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:13:10.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:13:10.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:13:10.817 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:13:10.817 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:13:10.817 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:13:10.817 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:13:10.817 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:13:10.817 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:13:10.818 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:13:10.818 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:10.818 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:10.818 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:13:10.818 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:13:10.818 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:13:10.906 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:13:10.906 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:13:10.907 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:13:10.908 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:13:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_repair: run_mgr td/ceph-helpers x 2024-05-06T04:13:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:13:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:13:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:13:10.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:13:10.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:13:10.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:13:11.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:13:11.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:13:11.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:13:11.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:13:11.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:11.839 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:11.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:13:11.839 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:13:11.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_repair: run_osd td/ceph-helpers 0 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:13:11.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:13:11.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:13:11.871 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:13:11.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31abdd26-f706-48c3-9967-1c8f174837fe 2024-05-06T04:13:11.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 31abdd26-f706-48c3-9967-1c8f174837fe' 2024-05-06T04:13:11.873 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 31abdd26-f706-48c3-9967-1c8f174837fe 2024-05-06T04:13:11.873 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:13:11.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXWDhmGMq/NBAAefe4OIlICZFhC6XrGJen0A== 2024-05-06T04:13:11.887 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXWDhmGMq/NBAAefe4OIlICZFhC6XrGJen0A=="}' 2024-05-06T04:13:11.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31abdd26-f706-48c3-9967-1c8f174837fe -i td/ceph-helpers/0/new.json 2024-05-06T04:13:12.173 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:13:12.183 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:13:12.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXWDhmGMq/NBAAefe4OIlICZFhC6XrGJen0A== --osd-uuid 31abdd26-f706-48c3-9967-1c8f174837fe 2024-05-06T04:13:12.213 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.210+0000 7f8279423640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:13:12.241 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.238+0000 7f8279423640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:13:12.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.239+0000 7f8279423640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:13:12.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.240+0000 7f8279423640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:13:12.243 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.240+0000 7f8279423640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:13:12.243 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.240+0000 7f8279423640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:13:12.281 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:12.278+0000 7f8279423640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:13:17.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:13:17.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:13:17.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:13:17.778 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:13:17.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:13:18.373 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:13:18.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:13:18.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:13:18.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:13:18.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:13:18.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:13:18.399 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:18.396+0000 7fb75c2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:13:18.401 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:18.398+0000 7fb75c2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:13:18.402 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:18.399+0000 7fb75c2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:13:18.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:13:18.799 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:13:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:13:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:13:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:13:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:18.800 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:13:18.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:13:18.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:18.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:18.920 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:18.918+0000 7fb75c2d9640 -1 Falling back to public interface 2024-05-06T04:13:19.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:20.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:20.227 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:13:20.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:20.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:13:20.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:20.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:20.659 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:21.254 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:13:21.251+0000 7fb75c2d9640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:13:21.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:21.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:21.661 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:13:21.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:13:21.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:21.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:22.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:23.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:23.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:23.316 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:13:23.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:13:23.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:23.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:23.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:24.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:24.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:24.747 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:13:24.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:13:24.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:24.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:25.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:26.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:26.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:26.184 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:13:26.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:13:26.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:26.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:26.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:27.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:27.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:27.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:13:27.594 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:13:27.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:27.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:28.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:29.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:29.117 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:29.117 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:13:29.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:13:29.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:29.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:29.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:30.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:30.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:30.551 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:13:30.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:13:30.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:30.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:30.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:31.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:31.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:31.973 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:13:31.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:13:31.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:31.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:32.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:33.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:33.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:33.417 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:13:33.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:13:33.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:33.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:33.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:34.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:34.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:34.944 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:13:34.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:13:34.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:34.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:35.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:36.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:36.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:36.584 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:13:36.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:13:36.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:36.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:37.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:38.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:38.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:38.370 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:13:38.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:13:38.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:38.371 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:39.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:40.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:40.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:40.097 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:13:40.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:13:40.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:40.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:40.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:41.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:41.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:41.548 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:13:41.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:13:41.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:41.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:42.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:13:43.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:13:43.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:13:43.666 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:13:43.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:13:43.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:13:43.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:13:44.137 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/652245293,v1:127.0.0.1:6803/652245293] [v2:127.0.0.1:6804/652245293,v1:127.0.0.1:6805/652245293] exists,up 31abdd26-f706-48c3-9967-1c8f174837fe 2024-05-06T04:13:44.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:13:44.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:13:44.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:13:44.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_repair: create_rbd_pool 2024-05-06T04:13:44.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:13:44.559 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:13:44.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:13:44.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:13:45.261 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:13:45.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:13:46.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:13:47.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_repair: wait_for_clean 2024-05-06T04:13:47.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:13:47.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:13:47.313 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:13:47.314 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:13:47.314 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:13:47.314 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:13:47.315 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:13:47.315 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:13:47.315 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:13:47.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:13:47.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:13:47.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:13:47.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:13:47.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:13:47.405 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:13:47.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:13:47.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:13:47.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:13:47.831 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:13:47.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836485 2024-05-06T04:13:47.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836485 2024-05-06T04:13:47.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836485' 2024-05-06T04:13:47.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:13:47.947 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836485 2024-05-06T04:13:47.947 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:13:47.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:13:47.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836485 2024-05-06T04:13:47.948 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:13:47.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836485 2024-05-06T04:13:47.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-05-06T04:13:47.950 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836485 2024-05-06T04:13:47.950 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:13:48.408 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836485 2024-05-06T04:13:48.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:13:48.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:13:48.408 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:13:48.944 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:13:49.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:13:49.357 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:13:49.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:13:49.358 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:13:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:13:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:13:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_repair: repair 1.0 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:49.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:50.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:50.312 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-05-06T04:13:50.689 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to repair 2024-05-06T04:13:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:13:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:13:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:13:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:50.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:50.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:50.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:50.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:50.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:51.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:44.894701+0000 '>' 2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:51.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:52.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:52.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:52.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:52.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:52.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:52.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:52.096 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:52.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:44.894701+0000 '>' 2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:52.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:53.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:53.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:53.489 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:53.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:53.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:53.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:53.490 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:53.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:44.894701+0000 '>' 2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:53.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:54.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:54.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:54.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:54.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:54.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:54.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:54.892 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:55.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:44.894701+0000 '>' 2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:55.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:56.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:56.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:13:56.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:56.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:56.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:56.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:56.294 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:50.872336+0000 '>' 2024-05-06T04:13:44.894701+0000 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:13:56.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:13:56.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:13:56.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:13:56.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: TIMEOUT=1 2024-05-06T04:13:56.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_repair: repair 1.0 2024-05-06T04:13:56.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: repair: local pgid=1.0 2024-05-06T04:13:56.793 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: get_last_scrub_stamp 1.0 2024-05-06T04:13:56.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:56.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:56.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:56.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:57.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: repair: local last_scrub=2024-05-06T04:13:50.872336+0000 2024-05-06T04:13:57.177 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: repair: ceph pg repair 1.0 2024-05-06T04:13:57.551 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to repair 2024-05-06T04:13:57.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: repair: wait_for_scrub 1.0 2024-05-06T04:13:50.872336+0000 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:13:50.872336+0000 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:13:57.565 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:13:57.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:13:50.872336+0000 '>' 2024-05-06T04:13:50.872336+0000 2024-05-06T04:13:57.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:13:58.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:13:58.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:13:58.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-05-06T04:13:58.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_repair: teardown td/ceph-helpers 2024-05-06T04:13:58.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:13:58.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:13:58.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:13:58.951 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:13:58.951 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:13:58.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:13:58.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:13:58.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:13:59.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:13:59.057 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:13:59.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:13:59.058 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:13:59.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:13:59.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:13:59.059 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:13:59.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:13:59.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:13:59.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:13:59.061 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:13:59.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:13:59.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:13:59.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:13:59.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:13:59.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:13:59.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:13:59.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:13:59.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:13:59.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:13:59.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:13:59.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_run_in_background td/ceph-helpers 2024-05-06T04:13:59.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_run_in_background: local pids 2024-05-06T04:13:59.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: test_run_in_background: run_in_background pids sleep 1 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 135431"' 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 135431' 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_run_in_background: run_in_background pids sleep 1 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sleep 1 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 135433"' 2024-05-06T04:13:59.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 135433' 2024-05-06T04:13:59.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/135434: /' 2024-05-06T04:13:59.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sleep 1 2024-05-06T04:13:59.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/135437: /' 2024-05-06T04:13:59.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_run_in_background: echo 135431 135433 2024-05-06T04:13:59.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_run_in_background: wc -w 2024-05-06T04:13:59.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_run_in_background: test 2 = 2 2024-05-06T04:13:59.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_run_in_background: wait 135431 135433 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_run_mon td/ceph-helpers 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-05-06T04:14:00.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:14:00.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:14:00.076 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:14:00.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:14:00.077 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:14:00.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:14:00.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:14:00.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:14:00.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:14:00.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:14:00.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:14:00.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:14:00.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:14:00.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:14:00.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:14:00.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:14:00.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:00.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:00.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:14:00.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:14:00.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:14:00.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:14:00.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:14:00.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:00.086 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:00.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:14:00.088 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:14:00.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:14:00.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T04:14:00.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:14:00.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T04:14:00.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:14:00.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:14:00.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:14:00.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:14:00.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:14:00.755 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:14:00.756 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:14:00.756 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:14:00.756 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:14:00.756 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:14:00.756 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:00.757 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:00.757 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:14:00.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:14:00.757 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:14:00.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:14:00.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:14:00.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:14:00.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:14:00.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:14:00.845 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:14:00.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-05-06T04:14:00.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-05-06T04:14:01.717 INFO:tasks.workunit.client.0.smithi163.stderr:dumped monmap epoch 1 2024-05-06T04:14:01.726 INFO:tasks.workunit.client.0.smithi163.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-05-06T04:14:01.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-05-06T04:14:01.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:14:01.727 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:14:01.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:14:01.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:14:01.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:14:02.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:14:02.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-05-06T04:14:02.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:14:02.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:14:02.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:14:02.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:14:02.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:14:02.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-05-06T04:14:02.056 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.053+0000 7fc4d4e38b00 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-05-06T04:14:02.058 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:14:02.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-05-06T04:14:02.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:14:02.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:14:02.260 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:14:02.261 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:02.342 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:02.343 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:14:02.343 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:14:02.343 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:14:02.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-05-06T04:14:02.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:14:02.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:14:02.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:14:02.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=716c64a1-6a98-435e-a906-1a58e6767a89 2024-05-06T04:14:02.429 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 716c64a1-6a98-435e-a906-1a58e6767a89 2024-05-06T04:14:02.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 716c64a1-6a98-435e-a906-1a58e6767a89' 2024-05-06T04:14:02.430 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:14:02.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKWThmf01mGhAAJUxT5OHPVJXtrjdR0VDwZg== 2024-05-06T04:14:02.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKWThmf01mGhAAJUxT5OHPVJXtrjdR0VDwZg=="}' 2024-05-06T04:14:02.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 716c64a1-6a98-435e-a906-1a58e6767a89 -i td/ceph-helpers/0/new.json 2024-05-06T04:14:02.705 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:14:02.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:14:02.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKWThmf01mGhAAJUxT5OHPVJXtrjdR0VDwZg== --osd-uuid 716c64a1-6a98-435e-a906-1a58e6767a89 2024-05-06T04:14:02.743 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.740+0000 7fafac80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:02.773 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.770+0000 7fafac80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:02.774 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.771+0000 7fafac80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:02.774 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.772+0000 7fafac80f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:14:02.774 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.772+0000 7fafac80f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:14:02.775 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.772+0000 7fafac80f640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:14:02.813 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:02.810+0000 7fafac80f640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:14:08.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:14:08.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:14:08.017 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:14:08.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:14:08.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:14:08.223 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:14:08.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:14:08.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:14:08.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:14:08.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:14:08.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:14:08.249 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:08.246+0000 7f12c5cd7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:08.256 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:08.253+0000 7f12c5cd7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:08.257 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:08.254+0000 7f12c5cd7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:08.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:14:08.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:14:08.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:14:08.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:14:08.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:14:08.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:08.435 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:14:08.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:14:08.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:08.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:08.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:08.788 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:08.786+0000 7f12c5cd7640 -1 Falling back to public interface 2024-05-06T04:14:09.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:09.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:09.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:14:09.654 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:14:09.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:09.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:09.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:10.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:10.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:10.853 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:14:10.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:14:10.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:10.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:11.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:11.222 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:11.219+0000 7f12c5cd7640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:14:12.046 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:12.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:12.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:14:12.047 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:14:12.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:12.047 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:12.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:13.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:13.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:13.362 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:14:13.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:14:13.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:13.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:13.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:14.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:14.562 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:14.562 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:14:14.563 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:14:14.563 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:14.563 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:14.761 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:15.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:15.762 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:15.763 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:14:15.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:14:15.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:15.763 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:15.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:16.963 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:16.963 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:14:16.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:16.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:14:16.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:16.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:17.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:18.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:18.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:18.161 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:14:18.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:14:18.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:18.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:18.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:19.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:19.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:19.363 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:14:19.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:14:19.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:19.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:19.555 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:20.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:20.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:20.557 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:14:20.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:14:20.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:20.557 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:20.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:21.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:21.750 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:14:21.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:21.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:14:21.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:21.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:21.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:22.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:22.954 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:14:22.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:22.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:14:22.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:22.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:23.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:24.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:24.154 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:14:24.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:24.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:14:24.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:24.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:24.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:25.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:25.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:25.352 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:14:25.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:14:25.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:25.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:25.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:26.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:26.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:26.545 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:14:26.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:14:26.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:26.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:26.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:27.744 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:14:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:14:27.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:27.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:27.941 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:28.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:28.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:28.943 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:14:28.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:14:28.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:28.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:29.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:30.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:30.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:30.146 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:14:30.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:14:30.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:30.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:30.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:31.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:31.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:31.341 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:14:31.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:14:31.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:31.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:32.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:32.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:32.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:14:32.544 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:14:32.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:32.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:14:32.771 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4166570897,v1:127.0.0.1:6801/4166570897] [v2:127.0.0.1:6802/4166570897,v1:127.0.0.1:6803/4166570897] exists,up 716c64a1-6a98-435e-a906-1a58e6767a89 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:14:32.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:14:32.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:14:32.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:14:32.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5ccd599-8473-4a2b-bcd5-39eb3ea6e8dd 2024-05-06T04:14:32.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f5ccd599-8473-4a2b-bcd5-39eb3ea6e8dd' 2024-05-06T04:14:32.779 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 f5ccd599-8473-4a2b-bcd5-39eb3ea6e8dd 2024-05-06T04:14:32.779 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:14:32.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoWThmSzkeLxAAZQjAeCbnTPC6zvf05EmAFQ== 2024-05-06T04:14:32.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoWThmSzkeLxAAZQjAeCbnTPC6zvf05EmAFQ=="}' 2024-05-06T04:14:32.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5ccd599-8473-4a2b-bcd5-39eb3ea6e8dd -i td/ceph-helpers/1/new.json 2024-05-06T04:14:33.078 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:14:33.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:14:33.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoWThmSzkeLxAAZQjAeCbnTPC6zvf05EmAFQ== --osd-uuid f5ccd599-8473-4a2b-bcd5-39eb3ea6e8dd 2024-05-06T04:14:33.116 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.114+0000 7ffb1be10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:33.118 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.115+0000 7ffb1be10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:33.119 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.116+0000 7ffb1be10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:33.120 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.117+0000 7ffb1be10640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:14:33.120 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.117+0000 7ffb1be10640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:14:33.120 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.117+0000 7ffb1be10640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:14:33.187 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:33.184+0000 7ffb1be10640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:14:39.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:14:39.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:14:39.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:14:39.126 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:14:39.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:14:39.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:14:39.333 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:14:39.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:14:39.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:14:39.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:14:39.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:14:39.359 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:39.356+0000 7f77c62d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:39.364 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:39.361+0000 7f77c62d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:39.366 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:39.363+0000 7f77c62d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:14:39.543 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:14:39.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:39.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:39.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:39.900 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:39.898+0000 7f77c62d9640 -1 Falling back to public interface 2024-05-06T04:14:40.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:40.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:40.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:14:40.752 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:14:40.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:40.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:40.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:41.950 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:41.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:41.951 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:14:41.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:14:41.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:41.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:42.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:42.329 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:14:42.327+0000 7f77c62d9640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:14:43.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:43.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:43.150 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:14:43.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:14:43.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:43.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:43.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:44.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:44.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:44.457 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:14:44.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:14:44.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:44.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:44.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:45.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:45.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:45.660 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:14:45.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:14:45.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:45.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:45.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:46.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:46.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:46.865 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:14:46.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:14:46.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:46.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:47.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:48.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:48.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:48.067 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:14:48.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:14:48.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:48.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:48.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:49.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:49.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:49.266 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:14:49.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:14:49.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:49.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:49.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:50.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:50.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:50.465 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:14:50.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:14:50.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:50.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:50.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:51.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:51.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:51.665 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:14:51.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:14:51.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:51.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:51.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:52.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:52.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:52.865 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:14:52.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:14:52.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:52.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:53.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:54.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:54.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:54.067 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:14:54.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:14:54.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:54.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:54.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:55.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:55.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:55.260 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:14:55.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:14:55.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:55.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:55.459 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:56.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:56.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:56.461 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:14:56.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:14:56.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:56.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:56.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:57.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:57.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:57.660 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:14:57.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:14:57.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:57.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:57.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:14:58.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:14:58.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:14:58.860 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:14:58.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:14:58.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:14:58.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:14:59.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:00.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:00.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:00.055 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:15:00.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:15:00.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:00.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:15:00.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:01.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:01.255 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:01.256 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:15:01.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:15:01.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:01.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:15:01.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:02.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:02.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:02.455 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:15:02.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:15:02.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:02.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:15:02.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:03.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:03.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:03.655 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:15:03.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:15:03.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:03.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:15:03.852 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3105353701,v1:127.0.0.1:6809/3105353701] [v2:127.0.0.1:6810/3105353701,v1:127.0.0.1:6811/3105353701] exists,up f5ccd599-8473-4a2b-bcd5-39eb3ea6e8dd 2024-05-06T04:15:03.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:15:03.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:15:03.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:15:03.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:15:03.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:15:03.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:15:03.859 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 50d0c265-e93c-43a3-8e2c-3893751eb86b 2024-05-06T04:15:03.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50d0c265-e93c-43a3-8e2c-3893751eb86b 2024-05-06T04:15:03.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 50d0c265-e93c-43a3-8e2c-3893751eb86b' 2024-05-06T04:15:03.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:15:03.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHWThmq/zwMxAA73jTJwUT45NqxFwxwmFQTw== 2024-05-06T04:15:03.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHWThmq/zwMxAA73jTJwUT45NqxFwxwmFQTw=="}' 2024-05-06T04:15:03.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50d0c265-e93c-43a3-8e2c-3893751eb86b -i td/ceph-helpers/2/new.json 2024-05-06T04:15:04.164 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:15:04.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-05-06T04:15:04.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHWThmq/zwMxAA73jTJwUT45NqxFwxwmFQTw== --osd-uuid 50d0c265-e93c-43a3-8e2c-3893751eb86b 2024-05-06T04:15:04.203 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.199+0000 7f0d0062c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:04.204 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.201+0000 7f0d0062c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:04.205 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.202+0000 7f0d0062c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:04.206 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.203+0000 7f0d0062c640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:15:04.206 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.203+0000 7f0d0062c640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:15:04.206 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.203+0000 7f0d0062c640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:15:04.273 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:04.270+0000 7f0d0062c640 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-05-06T04:15:10.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-05-06T04:15:10.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:15:10.008 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:15:10.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:15:10.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:15:10.215 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:15:10.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:15:10.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:15:10.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:15:10.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:15:10.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:15:10.242 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:10.238+0000 7f9490133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:10.244 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:10.240+0000 7f9490133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:10.245 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:10.241+0000 7f9490133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:15:10.420 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:15:10.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:10.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:10.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:10.771 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:10.767+0000 7f9490133640 -1 Falling back to public interface 2024-05-06T04:15:11.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:11.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:11.623 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:15:11.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:15:11.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:11.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:11.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:12.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:12.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:12.814 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:15:12.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:15:12.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:12.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:13.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:13.208 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:13.205+0000 7f9490133640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:15:14.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:14.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:14.016 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:15:14.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:15:14.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:14.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:14.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:15.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:15.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:15.324 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:15:15.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:15:15.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:15.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:15.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:16.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:16.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:16.521 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:15:16.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:15:16.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:16.522 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:16.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:17.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:17.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:17.724 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:15:17.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:15:17.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:17.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:17.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:18.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:18.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:18.926 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:15:18.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:15:18.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:18.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:19.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:20.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:20.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:20.131 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:15:20.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:15:20.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:20.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:20.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:21.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:21.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:21.332 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:15:21.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:15:21.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:21.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:21.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:22.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:22.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:22.532 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:15:22.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:15:22.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:22.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:22.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:23.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:23.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:23.732 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:15:23.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:15:23.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:23.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:23.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:24.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:24.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:24.933 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:15:24.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:15:24.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:24.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:25.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:26.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:26.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:26.131 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:15:26.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:15:26.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:26.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:26.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:27.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:27.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:27.333 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:15:27.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:15:27.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:27.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:27.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:28.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:28.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:28.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:15:28.532 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:15:28.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:28.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:28.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:29.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:29.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:29.738 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:15:29.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:15:29.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:29.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:29.936 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:30.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:30.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:30.938 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:15:30.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:15:30.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:30.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:31.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:32.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:32.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:32.137 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:15:32.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:15:32.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:32.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:32.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:33.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:33.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:33.337 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:15:33.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:15:33.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:33.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:33.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:34.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:34.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:34.538 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:15:34.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:15:34.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:34.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:15:34.741 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/231552054,v1:127.0.0.1:6817/231552054] [v2:127.0.0.1:6818/231552054,v1:127.0.0.1:6819/231552054] exists,up 50d0c265-e93c-43a3-8e2c-3893751eb86b 2024-05-06T04:15:34.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:15:34.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:15:34.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:15:34.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-05-06T04:15:34.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:15:34.950 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:15:34.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:15:34.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:15:35.290 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:15:35.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:15:36.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:15:38.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-05-06T04:15:38.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-05-06T04:15:38.257 INFO:tasks.workunit.client.0.smithi163.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 22 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2024-05-06T04:15:38.258 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-05-06T04:15:38.258 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:38.258 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:38.258 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:38.259 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:38.259 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:38.259 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:38.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-05-06T04:15:38.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get osd_pool_default_size 2024-05-06T04:15:38.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-05-06T04:15:38.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-05-06T04:15:38.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-05-06T04:15:38.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-05-06T04:15:38.407 INFO:tasks.workunit.client.0.smithi163.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-05-06T04:15:38.408 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.404+0000 7fd17191c640 -1 failed for service _ceph-mon._tcp 2024-05-06T04:15:38.408 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.404+0000 7fd17191c640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-05-06T04:15:38.408 INFO:tasks.workunit.client.0.smithi163.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-05-06T04:15:38.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-05-06T04:15:38.409 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-05-06T04:15:38.481 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.478+0000 7fd06059f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-05-06T04:15:38.481 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.478+0000 7fd06059f640 -1 AuthRegistry(0x7fd058069990) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-05-06T04:15:38.481 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.479+0000 7fd06059f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-05-06T04:15:38.481 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.479+0000 7fd06059f640 -1 AuthRegistry(0x7fd06059e0b0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-05-06T04:15:38.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.480+0000 7fd06059f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-05-06T04:15:38.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.480+0000 7fd06059f640 -1 AuthRegistry(0x7fd058066aa0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-05-06T04:15:38.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.480+0000 7fd06059f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-05-06T04:15:38.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.480+0000 7fd06059f640 -1 AuthRegistry(0x7fd0581032d0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-05-06T04:15:38.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.480+0000 7fd06059f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-05-06T04:15:38.483 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:38.480+0000 7fd06059f640 -1 AuthRegistry(0x7fd058066aa0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: cluster: 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: id: fc6d983f-746c-4b93-98c2-58f795783e09 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: health: HEALTH_OK 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: services: 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: mon: 1 daemons, quorum a (age 96s) 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: mgr: no daemons active 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: osd: 3 osds: 3 up (since 4s), 3 in (since 34s) 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: 2024-05-06T04:15:38.588 INFO:tasks.workunit.client.0.smithi163.stdout: data: 2024-05-06T04:15:38.589 INFO:tasks.workunit.client.0.smithi163.stdout: pools: 0 pools, 0 pgs 2024-05-06T04:15:38.589 INFO:tasks.workunit.client.0.smithi163.stdout: objects: 0 objects, 0 B 2024-05-06T04:15:38.589 INFO:tasks.workunit.client.0.smithi163.stdout: usage: 0 B used, 0 B / 0 B avail 2024-05-06T04:15:38.589 INFO:tasks.workunit.client.0.smithi163.stdout: pgs: 2024-05-06T04:15:38.589 INFO:tasks.workunit.client.0.smithi163.stdout: 2024-05-06T04:15:38.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-05-06T04:15:38.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:15:38.598 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:15:38.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:15:38.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:15:38.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:15:43.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:15:43.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:15:43.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:15:43.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:15:43.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:15:43.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:15:43.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:15:43.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:15:43.934 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:43.932+0000 7f9362c0cb00 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-05-06T04:15:43.937 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:15:43.937 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:15:43.937 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:15:43.937 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:15:43.937 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:43.937 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:43.938 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:43.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:15:44.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:15:44.148 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:15:44.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:15:44.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:15:44.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:15:44.149 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:44.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:15:44.151 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:44.236 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:44.237 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.237 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.237 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:44.237 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:15:44.237 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:15:44.318 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-05-06T04:15:44.318 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:44.318 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:44.319 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:44.319 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.319 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.319 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:44.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-05-06T04:15:44.319 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get osd_pool_default_size 2024-05-06T04:15:44.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-05-06T04:15:44.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-05-06T04:15:44.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-05-06T04:15:44.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:15:44.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:15:44.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:15:44.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:15:44.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:15:44.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T04:15:44.522 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:44.519+0000 7f0489211b00 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:44.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T04:15:44.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:15:44.687 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:15:44.687 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:15:44.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:15:44.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:15:44.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-05-06T04:15:44.848 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get osd_pool_default_size 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:15:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:15:45.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:15:45.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:15:45.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:15:45.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:15:45.036 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:15:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:15:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:15:45.038 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:15:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:15:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:15:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:15:45.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:15:45.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:15:45.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:15:45.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:15:45.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:15:45.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:45.051 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:45.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:15:45.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:15:45.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:15:45.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:15:45.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_run_osd td/ceph-helpers 2024-05-06T04:15:45.053 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:15:45.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:15:45.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:15:45.055 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:15:45.055 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:15:45.055 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:15:45.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:15:45.056 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:15:45.056 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:15:45.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:15:45.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:15:45.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:15:45.058 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:15:45.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:15:45.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:15:45.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:15:45.062 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:15:45.062 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:45.062 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:45.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:15:45.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:15:45.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:15:45.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:15:45.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:15:45.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:45.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:45.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:15:45.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:15:45.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:15:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:15:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:15:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-05-06T04:15:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:15:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:15:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:15:45.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:15:45.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:15:45.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-05-06T04:15:45.539 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:15:45.539 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:15:45.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:15:45.540 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:15:45.540 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:45.540 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:45.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:45.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-06T04:15:45.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:15:45.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:15:45.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:15:45.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:15:45.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:15:45.725 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:15:45.725 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:15:45.726 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:15:45.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:15:45.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:15:45.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:15:45.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:15:45.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:15:45.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:15:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:46.737 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:15:46.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:15:46.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-05-06T04:15:46.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:15:46.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:15:46.769 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:15:46.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f8078f3-a5d0-4b8d-9c6f-1c9b3611cc05 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2f8078f3-a5d0-4b8d-9c6f-1c9b3611cc05' 2024-05-06T04:15:46.771 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 2f8078f3-a5d0-4b8d-9c6f-1c9b3611cc05 2024-05-06T04:15:46.772 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:15:46.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByWThm3x21LhAAVp/D2/+xTgMY2QXoTqdw+g== 2024-05-06T04:15:46.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByWThm3x21LhAAVp/D2/+xTgMY2QXoTqdw+g=="}' 2024-05-06T04:15:46.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f8078f3-a5d0-4b8d-9c6f-1c9b3611cc05 -i td/ceph-helpers/0/new.json 2024-05-06T04:15:47.065 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:15:47.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:15:47.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByWThm3x21LhAAVp/D2/+xTgMY2QXoTqdw+g== --osd-uuid 2f8078f3-a5d0-4b8d-9c6f-1c9b3611cc05 2024-05-06T04:15:47.104 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.102+0000 7faa1620e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:47.133 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.130+0000 7faa1620e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:47.134 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.131+0000 7faa1620e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:47.135 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.132+0000 7faa1620e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:15:47.135 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.132+0000 7faa1620e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:15:47.135 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.132+0000 7faa1620e640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:15:47.173 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:47.170+0000 7faa1620e640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:15:52.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:15:52.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:15:52.738 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:15:52.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:15:52.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:15:53.323 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:15:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:15:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:15:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:15:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:15:53.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:15:53.349 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:53.345+0000 7f3e3f013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:53.364 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:53.360+0000 7f3e3f013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:53.365 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:53.361+0000 7f3e3f013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:15:53.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:15:53.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:15:53.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:15:53.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:15:53.786 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:15:53.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:15:53.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:53.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:15:53.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:53.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:15:53.903 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:53.900+0000 7f3e3f013640 -1 Falling back to public interface 2024-05-06T04:15:54.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:55.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:55.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:55.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:15:55.230 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:15:55.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:55.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:15:55.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:56.266 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:15:56.263+0000 7f3e3f013640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:15:56.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:56.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:56.656 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:15:56.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:15:56.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:56.657 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:15:57.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:58.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:58.238 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:15:58.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:58.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:15:58.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:58.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:15:58.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:15:59.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:15:59.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:15:59.683 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:15:59.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:15:59.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:15:59.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:00.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:01.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:01.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:01.163 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:16:01.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:16:01.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:01.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:01.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:02.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:02.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:02.571 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:16:02.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:16:02.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:02.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:03.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:04.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:04.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:04.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:16:04.018 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:16:04.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:04.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:04.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:05.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:05.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:05.432 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:16:05.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:16:05.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:05.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:05.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:06.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:06.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:06.838 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:16:06.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:16:06.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:06.838 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:07.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:08.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:08.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:08.357 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:16:08.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:16:08.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:08.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:09.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:10.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:10.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:10.067 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:16:10.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:16:10.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:10.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:10.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:11.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:11.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:11.663 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:16:11.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:16:11.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:11.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:12.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:13.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:13.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:13.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:16:13.375 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:16:13.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:13.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:14.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:15.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:15.162 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:15.162 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:16:15.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:16:15.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:15.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:15.856 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:16.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:16.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:16.858 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:16:16.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:16:16.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:16.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:17.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:18.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:18.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:18.598 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:16:18.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:16:18.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:18.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:16:19.002 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/647159202,v1:127.0.0.1:6803/647159202] [v2:127.0.0.1:6804/647159202,v1:127.0.0.1:6805/647159202] exists,up 2f8078f3-a5d0-4b8d-9c6f-1c9b3611cc05 2024-05-06T04:16:19.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:16:19.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:16:19.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.0.asok 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-05-06T04:16:19.005 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.0.asok config get osd_max_backfills 2024-05-06T04:16:19.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-05-06T04:16:19.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-05-06T04:16:19.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:16:19.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:16:19.096 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:16:19.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b772da35-949b-4629-8bae-5ef23b40bb7d 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b772da35-949b-4629-8bae-5ef23b40bb7d' 2024-05-06T04:16:19.098 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 b772da35-949b-4629-8bae-5ef23b40bb7d 2024-05-06T04:16:19.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:16:19.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTWThmuoeLBhAAd9CBgh8QOm7rptvLPXbClg== 2024-05-06T04:16:19.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTWThmuoeLBhAAd9CBgh8QOm7rptvLPXbClg=="}' 2024-05-06T04:16:19.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b772da35-949b-4629-8bae-5ef23b40bb7d -i td/ceph-helpers/1/new.json 2024-05-06T04:16:19.819 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:16:19.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:16:19.831 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQCTWThmuoeLBhAAd9CBgh8QOm7rptvLPXbClg== --osd-uuid b772da35-949b-4629-8bae-5ef23b40bb7d 2024-05-06T04:16:19.859 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.856+0000 7fe14760f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:19.894 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.891+0000 7fe14760f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:19.895 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.892+0000 7fe14760f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:19.895 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.892+0000 7fe14760f640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:16:19.895 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.892+0000 7fe14760f640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:16:19.895 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.892+0000 7fe14760f640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:16:19.977 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:19.974+0000 7fe14760f640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:16:25.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:16:25.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:16:25.788 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:16:25.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:16:25.789 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:16:26.358 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:16:26.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:16:26.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-05-06T04:16:26.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:16:26.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:16:26.359 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:16:26.385 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:26.382+0000 7f82485dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:26.388 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:26.386+0000 7f82485dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:26.389 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:26.387+0000 7f82485dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:26.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:16:26.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:16:26.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:16:26.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:16:26.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:16:26.804 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:16:26.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:26.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:16:26.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:26.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:26.921 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:26.919+0000 7f82485dc640 -1 Falling back to public interface 2024-05-06T04:16:27.204 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:28.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:28.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:28.206 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:16:28.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:16:28.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:28.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:28.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:29.334 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:29.331+0000 7f82485dc640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:16:29.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:29.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:29.649 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:16:29.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:16:29.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:29.649 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:30.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:31.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:31.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:31.134 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:16:31.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:16:31.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:31.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:31.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:32.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:32.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:32.539 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:16:32.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:16:32.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:32.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:33.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:33.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:33.944 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:16:33.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:16:33.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:33.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:34.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:35.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:35.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:35.473 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:16:35.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:16:35.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:35.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:35.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:36.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:36.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:36.896 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:16:36.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:16:36.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:36.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:37.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:38.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:38.335 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:38.336 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:16:38.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:16:38.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:38.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:38.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:39.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:39.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:39.739 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:16:39.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:16:39.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:39.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:40.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:41.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:16:41.282 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:16:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:41.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:41.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:42.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:42.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:42.905 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:16:42.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:16:42.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:42.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:43.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:44.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:44.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:44.476 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:16:44.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:16:44.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:44.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:45.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:46.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:46.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:46.245 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:16:46.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:16:46.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:46.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:47.279 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:48.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:48.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:48.281 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:16:48.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:16:48.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:48.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:49.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:50.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:50.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:50.133 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:16:50.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:16:50.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:50.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:50.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:51.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:51.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:51.945 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:16:51.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:16:51.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:51.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:52.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:16:53.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:16:53.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:16:53.511 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:16:53.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:16:53.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:16:53.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:16:53.936 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1942605512,v1:127.0.0.1:6811/1942605512] [v2:127.0.0.1:6812/1942605512,v1:127.0.0.1:6813/1942605512] exists,up b772da35-949b-4629-8bae-5ef23b40bb7d 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:16:53.938 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.1.asok 2024-05-06T04:16:53.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-05-06T04:16:53.939 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.1.asok config get osd_max_backfills 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:16:54.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:16:54.027 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:16:54.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:16:54.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:16:54.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:16:54.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-05-06T04:16:54.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:16:54.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=971a21f1-04ee-4bc0-b6fa-7536de966a96 2024-05-06T04:16:54.030 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 971a21f1-04ee-4bc0-b6fa-7536de966a96 2024-05-06T04:16:54.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 971a21f1-04ee-4bc0-b6fa-7536de966a96' 2024-05-06T04:16:54.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:16:54.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2WThmGOGHAhAAqecFyxkEyDWcOWFnIBVliw== 2024-05-06T04:16:54.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2WThmGOGHAhAAqecFyxkEyDWcOWFnIBVliw=="}' 2024-05-06T04:16:54.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 971a21f1-04ee-4bc0-b6fa-7536de966a96 -i td/ceph-helpers/2/new.json 2024-05-06T04:16:54.496 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:16:54.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-05-06T04:16:54.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2WThmGOGHAhAAqecFyxkEyDWcOWFnIBVliw== --osd-uuid 971a21f1-04ee-4bc0-b6fa-7536de966a96 2024-05-06T04:16:54.537 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.535+0000 7f9915c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:54.578 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.575+0000 7f9915c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:54.579 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.576+0000 7f9915c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:16:54.579 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.576+0000 7f9915c26640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:16:54.579 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.576+0000 7f9915c26640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:16:54.579 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.576+0000 7f9915c26640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:16:54.618 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:16:54.615+0000 7f9915c26640 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-05-06T04:17:00.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-05-06T04:17:00.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:17:00.717 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:17:00.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:17:00.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:17:01.270 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:17:01.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:17:01.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:17:01.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:17:01.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:17:01.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:17:01.299 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:01.296+0000 7fac50613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:01.301 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:01.298+0000 7fac50613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:01.302 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:01.300+0000 7fac50613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:01.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:17:01.741 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:17:01.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:01.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:01.829 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:01.827+0000 7fac50613640 -1 Falling back to public interface 2024-05-06T04:17:02.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:03.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:03.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:03.175 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:17:03.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:17:03.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:03.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:03.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:04.313 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:04.310+0000 7fac50613640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:17:04.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:04.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:04.610 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:17:04.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:17:04.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:04.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:05.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:06.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:06.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:06.266 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:17:06.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:17:06.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:06.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:06.766 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:07.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:07.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:07.767 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:17:07.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:17:07.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:07.768 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:08.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:09.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:09.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:09.214 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:17:09.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:17:09.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:09.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:09.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:10.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:10.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:10.620 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:17:10.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:17:10.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:10.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:11.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:12.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:12.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:12.211 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:17:12.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:17:12.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:12.211 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:12.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:13.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:13.618 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:17:13.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:13.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:17:13.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:13.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:14.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:15.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:15.109 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:15.109 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:17:15.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:17:15.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:15.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:15.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:16.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:16.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:16.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:17:16.534 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:17:16.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:16.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:16.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:17.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:17.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:17.942 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:17:17.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:17:17.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:17.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:18.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:19.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:19.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:19.972 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:17:19.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:17:19.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:19.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:20.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:21.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:21.448 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:17:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:17:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:22.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:23.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:23.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:23.353 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:17:23.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:17:23.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:23.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:24.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:25.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:25.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:25.308 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:17:25.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:17:25.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:25.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:26.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:27.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:27.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:27.127 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:17:27.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:17:27.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:27.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:27.582 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:28.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:28.583 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:28.584 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:17:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:17:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:28.584 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:17:29.047 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1585111611,v1:127.0.0.1:6819/1585111611] [v2:127.0.0.1:6820/1585111611,v1:127.0.0.1:6821/1585111611] exists,up 971a21f1-04ee-4bc0-b6fa-7536de966a96 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-osd.2.asok 2024-05-06T04:17:29.049 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-05-06T04:17:29.050 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.75593/ceph-osd.2.asok config get osd_max_backfills 2024-05-06T04:17:29.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-05-06T04:17:29.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:29.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:29.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:29.242 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:29.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:29.243 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:29.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:29.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:29.244 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:29.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:29.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:29.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:29.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:29.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:29.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:29.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:29.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:29.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:29.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:29.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_set_config td/ceph-helpers 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:29.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:29.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:29.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:29.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:29.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:29.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:29.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:29.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:29.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:29.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:29.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:29.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:29.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:29.264 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:29.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:29.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:29.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:29.266 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:29.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:29.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:29.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:29.268 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:29.268 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:29.268 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:29.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:29.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:29.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:29.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:17:29.270 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:17:29.270 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:29.271 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:29.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:17:29.272 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:17:29.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:17:29.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:17:29.274 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:17:29.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:17:29.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:29.931 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:29.932 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:29.932 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:29.932 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:29.932 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:17:30.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:17:30.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:30.011 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:17:30.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-05-06T04:17:30.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:30.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:30.091 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:30.092 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get ms_crc_header 2024-05-06T04:17:30.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-05-06T04:17:30.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-05-06T04:17:30.170 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:30.171 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config set ms_crc_header false 2024-05-06T04:17:30.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:30.253 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.254 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.254 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:30.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:30.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get ms_crc_header 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:30.332 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:30.333 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:30.333 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.333 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.333 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:30.333 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config set ms_crc_header true 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:30.413 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.414 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.414 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:30.414 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:30.414 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get ms_crc_header 2024-05-06T04:17:30.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-05-06T04:17:30.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-05-06T04:17:30.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:30.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:30.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:30.494 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:30.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:30.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:30.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:30.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:30.597 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:30.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:30.597 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:30.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:30.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:30.599 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:30.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:30.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:30.600 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:30.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:30.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:30.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:30.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_setup td/ceph-helpers 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:30.606 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:30.607 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:30.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:30.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:30.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:30.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:30.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:30.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:30.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:30.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:30.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:30.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:30.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:30.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:30.610 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:30.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:30.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:30.612 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:30.612 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.612 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:30.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:30.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:30.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:17:30.614 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:17:30.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:17:30.617 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:17:30.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:17:30.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:17:30.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:17:30.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:30.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:30.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:30.619 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:30.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:30.620 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:30.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:30.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:30.620 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:30.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:30.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:30.622 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:30.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:30.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:30.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:30.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.624 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:30.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:30.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:30.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:17:30.626 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:17:30.626 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.627 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:30.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:30.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:30.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:30.630 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:30.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:30.631 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:30.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:30.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:30.632 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:30.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:30.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:30.634 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:30.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:30.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:30.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:30.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.636 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:30.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:30.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:30.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_teardown td/ceph-helpers 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:30.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:30.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:30.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:30.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:30.640 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:30.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:30.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:30.642 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:30.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:30.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:30.643 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:30.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:30.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:30.645 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:30.646 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.646 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:30.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:30.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:30.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:17:30.648 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:17:30.648 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.648 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:30.650 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:30.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:30.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:30.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:30.651 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:30.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:30.652 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:30.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:30.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:30.653 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:30.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:30.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:30.655 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:30.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:30.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:30.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:30.660 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_background td/ceph-helpers 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: test_wait_background: local pids= 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2198: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 147103"' 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 147103' 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 1; exit 1' 2024-05-06T04:17:30.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 147105"' 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 147105' 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_wait_background: wait_background pids 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/147106: /' 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 147103 147105' 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 147103 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 2; exit 0' 2024-05-06T04:17:30.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/147108: /' 2024-05-06T04:17:31.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 1 2024-05-06T04:17:31.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: wait_background: return_code=1 2024-05-06T04:17:31.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-05-06T04:17:31.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 147105 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 1 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_wait_background: '[' 1 -ne 1 ']' 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 147111"' 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 147111' 2024-05-06T04:17:32.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2204: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: run_in_background: local pid_variable=pids 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: run_in_background: shift 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 1; exit 0' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: eval 'pids+=" 147114"' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: run_in_background: pids+=' 147114' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/147113: /' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: test_wait_background: wait_background pids 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: wait_background: pids=' 147111 147114' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: wait_background: return_code=0 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 147111 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: bash -c 'sleep 2; exit 0' 2024-05-06T04:17:32.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: sed 's/^/147116: /' 2024-05-06T04:17:33.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-05-06T04:17:33.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: for pid in $pids 2024-05-06T04:17:33.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: wait_background: wait 147114 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: run_in_background: return 0 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: eval 'pids='\'''\''' 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: wait_background: pids= 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: wait_background: return 0 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_wait_background: '[' 0 -ne 0 ']' 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_wait_background: '[' '!' -z '' ']' 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_clean td/ceph-helpers 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:17:34.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:17:34.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:17:34.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:17:34.669 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:17:34.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:17:34.669 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:17:34.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:17:34.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:17:34.671 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:17:34.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:34.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:17:34.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:17:34.672 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:17:34.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:17:34.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:17:34.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:17:34.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:17:34.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:34.676 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:34.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:17:34.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:17:34.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:17:34.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:17:34.678 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:17:34.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:34.679 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:34.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:17:34.680 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:17:34.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-05-06T04:17:35.142 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:17:35.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-05-06T04:17:35.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:17:35.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:17:35.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:35.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:35.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:17:35.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:35.355 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:17:35.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:17:35.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:17:35.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:17:35.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:17:35.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:17:35.445 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:17:35.446 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:17:35.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:17:35.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:17:35.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:17:35.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:17:35.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:17:35.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:17:35.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=253ed34d-4be5-41c2-b7ba-ed5c5b15df88 2024-05-06T04:17:35.532 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 253ed34d-4be5-41c2-b7ba-ed5c5b15df88 2024-05-06T04:17:35.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 253ed34d-4be5-41c2-b7ba-ed5c5b15df88' 2024-05-06T04:17:35.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:17:35.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfWThml6hmIBAAAAO318q545sd3c6PurRs3Q== 2024-05-06T04:17:35.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfWThml6hmIBAAAAO318q545sd3c6PurRs3Q=="}' 2024-05-06T04:17:35.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 253ed34d-4be5-41c2-b7ba-ed5c5b15df88 -i td/ceph-helpers/0/new.json 2024-05-06T04:17:36.400 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:17:36.410 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:17:36.411 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfWThml6hmIBAAAAO318q545sd3c6PurRs3Q== --osd-uuid 253ed34d-4be5-41c2-b7ba-ed5c5b15df88 2024-05-06T04:17:36.439 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.436+0000 7f8ed0311640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:36.469 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.466+0000 7f8ed0311640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:36.469 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.467+0000 7f8ed0311640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:36.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.467+0000 7f8ed0311640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:17:36.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.467+0000 7f8ed0311640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:17:36.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.467+0000 7f8ed0311640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:17:36.501 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:36.498+0000 7f8ed0311640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:17:41.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:17:41.948 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:17:41.948 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:17:41.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:17:41.949 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:17:42.156 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:17:42.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:17:42.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:17:42.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:17:42.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:17:42.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:17:42.183 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:42.180+0000 7f0ab5f16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:42.187 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:42.184+0000 7f0ab5f16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:42.188 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:42.186+0000 7f0ab5f16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:17:42.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:17:42.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:17:42.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:17:42.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:17:42.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:17:42.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:42.365 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:17:42.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:17:42.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:42.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:42.575 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:42.717 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:42.714+0000 7f0ab5f16640 -1 Falling back to public interface 2024-05-06T04:17:43.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:43.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:43.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:17:43.577 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:17:43.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:43.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:43.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:44.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:44.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:44.780 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:17:44.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:17:44.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:44.780 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:44.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:45.112 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:17:45.109+0000 7f0ab5f16640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:17:45.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:45.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:45.981 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:17:45.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:17:45.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:45.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:46.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:47.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:47.258 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:17:47.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:47.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:17:47.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:47.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:47.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:48.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:48.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:48.462 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:17:48.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:17:48.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:48.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:48.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:49.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:49.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:49.663 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:17:49.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:17:49.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:49.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:49.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:50.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:50.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:50.866 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:17:50.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:17:50.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:50.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:51.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:52.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:52.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:52.068 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:17:52.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:17:52.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:52.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:52.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:53.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:53.266 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:53.266 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:17:53.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:17:53.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:53.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:53.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:54.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:54.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:54.464 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:17:54.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:17:54.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:54.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:54.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:55.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:55.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:55.668 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:17:55.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:17:55.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:55.668 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:55.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:56.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:56.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:56.866 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:17:56.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:17:56.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:56.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:57.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:58.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:58.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:58.065 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:17:58.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:17:58.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:58.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:58.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:17:59.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:17:59.269 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:17:59.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:17:59.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:17:59.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:17:59.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:17:59.472 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:00.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:00.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:00.475 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:18:00.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:18:00.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:00.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:18:00.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:01.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:01.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:01.675 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:18:01.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:18:01.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:01.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:18:01.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:02.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:02.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:02.871 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:18:02.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:18:02.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:02.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:18:03.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:04.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:04.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:04.070 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:18:04.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:18:04.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:04.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:18:04.269 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:05.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:05.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:05.271 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:18:05.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:18:05.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:05.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:18:05.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:06.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:06.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:06.468 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:18:06.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:18:06.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:06.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:18:06.769 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3823974372,v1:127.0.0.1:6801/3823974372] [v2:127.0.0.1:6802/3823974372,v1:127.0.0.1:6803/3823974372] exists,up 253ed34d-4be5-41c2-b7ba-ed5c5b15df88 2024-05-06T04:18:06.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:18:06.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:18:06.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:18:06.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-05-06T04:18:06.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:18:06.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:18:06.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:18:06.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:18:06.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:18:06.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:18:07.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:18:07.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:18:07.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:18:07.101 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:18:07.101 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:18:07.101 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:18:07.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:18:07.102 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:18:07.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:18:07.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-05-06T04:18:07.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:18:07.318 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:18:07.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:18:07.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:18:07.722 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:18:07.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:18:08.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:18:09.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-05-06T04:18:09.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-05-06T04:18:09.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:18:09.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:18:09.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:18:09.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-05-06T04:18:09.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:18:09.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:18:09.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:18:09.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:18:09.801 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:18:09.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:18:09.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:18:09.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:18:09.996 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:18:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=0 2024-05-06T04:18:10.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 0 2024-05-06T04:18:10.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-0' 2024-05-06T04:18:10.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:18:10.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-0 2024-05-06T04:18:10.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:18:10.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:18:10.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-0 2024-05-06T04:18:10.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:18:10.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=0 2024-05-06T04:18:10.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-05-06T04:18:10.131 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 0 2024-05-06T04:18:10.131 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:18:10.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 0 -lt 0 2024-05-06T04:18:10.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:18:10.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:10.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:10.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-05-06T04:18:10.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-05-06T04:18:11.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:18:11.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:11.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:12.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:18:12.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:18:12.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:18:12.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:18:12.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:18:12.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:18:12.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:18:12.083 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:18:12.469 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:18:12.469 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:18:12.469 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:12.470 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:13.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:18:13.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-05-06T04:18:13.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-06T04:18:13.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-05-06T04:18:13.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:18:13.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-06T04:18:13.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:18:13.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:18:13.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:18:13.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:18:13.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:18:13.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:18:13.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:18:13.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:18:13.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:18:13.513 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:18:13.513 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:13.514 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:18:14.045 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:18:14.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:18:14.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:18:14.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-05-06T04:18:14.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:18:14.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-05-06T04:18:14.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:18:14.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:18:14.810 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:18:14.810 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:18:14.810 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:18:14.810 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:18:14.811 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:18:14.811 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:18:15.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:18:15.192 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:18:15.192 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:15.192 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:18:15.749 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:18:16.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:18:16.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:18:16.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-05-06T04:18:16.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:18:16.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:18:16.689 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:18:17.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:18:17.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:18:17.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:17.074 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:17.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:18:17.616 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:18:18.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:18:18.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:18:18.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-05-06T04:18:18.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:18:18.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-05-06T04:18:18.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:18:18.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:18:18.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:18:18.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:18:18.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:18:18.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:18:18.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:18:18.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:18:18.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:18:18.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:18:18.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:18.844 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:18:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:18:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:18:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:18:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:18:19.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:18:19.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:18:19.379 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:18:19.379 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:18:19.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:18:19.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:18:19.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-05-06T04:18:19.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-05-06T04:18:20.467 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T04:18:20.467 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_fingerprint": "a16766e1-e113-4473-a7f1-879f3efb8f22", 2024-05-06T04:18:20.467 INFO:tasks.workunit.client.0.smithi163.stdout: "version": "19.0.0-3516-ge08cbbf6", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "commit": "e08cbbf6fd79a8dd2949f340bce25e28ea3db004", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "timestamp": "2024-05-06T04:18:20.464258+0000", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "tag": "", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "health": { 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "status": "HEALTH_WARN", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "checks": { 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "PG_DEGRADED": { 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "severity": "HEALTH_WARN", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "summary": { 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "message": "Degraded data redundancy: 1/2 objects degraded (50.000%), 1 pg degraded", 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: "count": 1 2024-05-06T04:18:20.468 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "detail": [ 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "muted": false 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "TOO_FEW_OSDS": { 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "severity": "HEALTH_WARN", 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "summary": { 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-05-06T04:18:20.469 INFO:tasks.workunit.client.0.smithi163.stdout: "count": 2 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "detail": [], 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "muted": false 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "mutes": [] 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "monmap_first_committed": 1, 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "monmap_last_committed": 1, 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "monmap": { 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 1, 2024-05-06T04:18:20.470 INFO:tasks.workunit.client.0.smithi163.stdout: "fsid": "fc6d983f-746c-4b93-98c2-58f795783e09", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "modified": "2024-05-06T04:17:34.703280Z", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "created": "2024-05-06T04:17:34.703280Z", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "min_mon_release": 19, 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "min_mon_release_name": "squid", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "election_strategy": 1, 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "disallowed_leaders: ": "", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode": false, 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "tiebreaker_mon": "", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "removed_ranks: ": "", 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "features": { 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "persistent": [ 2024-05-06T04:18:20.471 INFO:tasks.workunit.client.0.smithi163.stdout: "kraken", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "luminous", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "mimic", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap-prune", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "nautilus", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "octopus", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "pacific", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "elector-pinging", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "quincy", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "reef", 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "squid" 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.472 INFO:tasks.workunit.client.0.smithi163.stdout: "optional": [] 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "mons": [ 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "rank": 0, 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "a", 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addrs": { 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:7109", 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 0 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.473 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:7109/0", 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addr": "127.0.0.1:7109/0", 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: "priority": 0, 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 0, 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_location": "{}" 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: "quorum": [ 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: 0 2024-05-06T04:18:20.474 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap": { 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 12, 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "fsid": "fc6d983f-746c-4b93-98c2-58f795783e09", 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "created": "2024-05-06T04:17:35.662912+0000", 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "modified": "2024-05-06T04:18:09.502870+0000", 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "last_up_change": "2024-05-06T04:18:06.624691+0000", 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "last_in_change": "2024-05-06T04:17:36.300930+0000", 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "flags_num": 5799936, 2024-05-06T04:18:20.475 INFO:tasks.workunit.client.0.smithi163.stdout: "flags_set": [ 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "pglog_hardlimit", 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "purged_snapdirs", 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "recovery_deletes", 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "sortbitwise" 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_version": 3, 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "full_ratio": 0.99000000953674316, 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_snapshot": "", 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_max": 1, 2024-05-06T04:18:20.476 INFO:tasks.workunit.client.0.smithi163.stdout: "max_osd": 1, 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "require_min_compat_client": "luminous", 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "min_compat_client": "jewel", 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "require_osd_release": "squid", 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "allow_crimson": false, 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "pools": [ 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_name": "rbd", 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "create_time": "2024-05-06T04:18:07.541846+0000", 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "flags": 8193, 2024-05-06T04:18:20.477 INFO:tasks.workunit.client.0.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "type": 1, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "size": 2, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "min_size": 1, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_rule": 0, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "object_hash": 2, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_autoscale_mode": "off", 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_num": 4, 2024-05-06T04:18:20.478 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_placement_num": 4, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_placement_num_target": 4, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_num_target": 4, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_num_pending": 4, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "last_pg_merge_meta": { 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "source_pgid": "0.0", 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "ready_epoch": 0, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "last_epoch_started": 0, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "last_epoch_clean": 0, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "source_version": "0'0", 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "target_version": "0'0" 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.479 INFO:tasks.workunit.client.0.smithi163.stdout: "last_change": "12", 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "last_force_op_resend": "0", 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "auid": 0, 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_mode": "selfmanaged", 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_seq": 2, 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_epoch": 12, 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_snaps": [], 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "removed_snaps": "[]", 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "quota_max_bytes": 0, 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "quota_max_objects": 0, 2024-05-06T04:18:20.480 INFO:tasks.workunit.client.0.smithi163.stdout: "tiers": [], 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "tier_of": -1, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "read_tier": -1, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "write_tier": -1, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_mode": "none", 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "target_max_bytes": 0, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "target_max_objects": 0, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_min_flush_age": 0, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_min_evict_age": 0, 2024-05-06T04:18:20.481 INFO:tasks.workunit.client.0.smithi163.stdout: "erasure_code_profile": "", 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_params": { 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "none" 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_period": 0, 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_count": 0, 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "use_gmt_hitset": true, 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-05-06T04:18:20.482 INFO:tasks.workunit.client.0.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_search_last_n": 0, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "grade_table": [], 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "stripe_width": 0, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "expected_num_objects": 0, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "fast_read": false, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "options": {}, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "application_metadata": { 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "rbd": {} 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "read_balance": { 2024-05-06T04:18:20.483 INFO:tasks.workunit.client.0.smithi163.stdout: "score_type": "Fair distribution", 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "score_acting": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "score_stable": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "optimal_score": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "raw_score_acting": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "raw_score_stable": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "primary_affinity_weighted": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "average_primary_affinity": 1, 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: "osds": [ 2024-05-06T04:18:20.484 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "uuid": "253ed34d-4be5-41c2-b7ba-ed5c5b15df88", 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "up": 1, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "in": 1, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 1, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "primary_affinity": 1, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "last_clean_begin": 0, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "last_clean_end": 0, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "up_from": 5, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "up_thru": 7, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "down_at": 0, 2024-05-06T04:18:20.485 INFO:tasks.workunit.client.0.smithi163.stdout: "lost_at": 0, 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addrs": { 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6800", 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6801", 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.486 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_addrs": { 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6802", 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6803", 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.487 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_back_addrs": { 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6806", 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6807", 2024-05-06T04:18:20.488 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_front_addrs": { 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6804", 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.489 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6805", 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 3823974372 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addr": "127.0.0.1:6801/3823974372", 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_addr": "127.0.0.1:6803/3823974372", 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_back_addr": "127.0.0.1:6807/3823974372", 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_front_addr": "127.0.0.1:6805/3823974372", 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "state": [ 2024-05-06T04:18:20.490 INFO:tasks.workunit.client.0.smithi163.stdout: "exists", 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "up" 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_xinfo": [ 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "down_stamp": "0.000000", 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "laggy_probability": 0, 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "laggy_interval": 0, 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "features": 4540701547738038271, 2024-05-06T04:18:20.491 INFO:tasks.workunit.client.0.smithi163.stdout: "old_weight": 0, 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "last_purged_snaps_scrub": "2024-05-06T04:18:06.322674+0000", 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "dead_epoch": 0 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_upmap": [], 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_upmap_items": [], 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_upmap_primaries": [], 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_temp": [], 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "primary_temp": [], 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "blocklist": {}, 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "range_blocklist": {}, 2024-05-06T04:18:20.492 INFO:tasks.workunit.client.0.smithi163.stdout: "erasure_code_profiles": { 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "default": { 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "k": "2", 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "m": "2", 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "plugin": "jerasure", 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "technique": "reed_sol_van" 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "removed_snaps_queue": [ 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "snaps": [ 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.493 INFO:tasks.workunit.client.0.smithi163.stdout: "begin": 2, 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: "length": 1 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: "new_removed_snaps": [ 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: "snaps": [ 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: "begin": 2, 2024-05-06T04:18:20.494 INFO:tasks.workunit.client.0.smithi163.stdout: "length": 1 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "new_purged_snaps": [], 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_node_flags": {}, 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "device_class_flags": {}, 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode": { 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode_enabled": false, 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_bucket_count": 0, 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "degraded_stretch_mode": 0, 2024-05-06T04:18:20.495 INFO:tasks.workunit.client.0.smithi163.stdout: "recovering_stretch_mode": 0, 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode_bucket": 0 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_metadata": [ 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "arch": "x86_64", 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "back_addr": "[v2:127.0.0.1:6802/3823974372,v1:127.0.0.1:6803/3823974372]", 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "back_iface": "", 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs": "1", 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs_dedicated_db": "0", 2024-05-06T04:18:20.496 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs_dedicated_wal": "0", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs_single_shared_device": "1", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_allocation_from_file": "0", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_access_mode": "file", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_block_size": "4096", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_devices": "sda", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_rotational": "1", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_size": "107374182400", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_support_discard": "1", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_type": "hdd", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_min_alloc_size": "4096", 2024-05-06T04:18:20.497 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_release": "squid", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_version": "ceph version 19.0.0-3516-ge08cbbf6 (e08cbbf6fd79a8dd2949f340bce25e28ea3db004) squid (dev)", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_version_short": "19.0.0-3516-ge08cbbf6", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_version_when_created": "ceph version 19.0.0-3516-ge08cbbf6 (e08cbbf6fd79a8dd2949f340bce25e28ea3db004) squid (dev)", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "created_at": "2024-05-06T04:17:41.275455Z", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "default_device_class": "hdd", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFMX", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "devices": "sda", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "distro": "centos", 2024-05-06T04:18:20.498 INFO:tasks.workunit.client.0.smithi163.stdout: "distro_description": "CentOS Stream 9", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "distro_version": "9", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "front_addr": "[v2:127.0.0.1:6800/3823974372,v1:127.0.0.1:6801/3823974372]", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "front_iface": "", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/3823974372,v1:127.0.0.1:6807/3823974372]", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/3823974372,v1:127.0.0.1:6805/3823974372]", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "hostname": "smithi163", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "journal_rotational": "1", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "kernel_version": "5.14.0-437.el9.x86_64", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "mem_swap_kb": "0", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "mem_total_kb": "32488420", 2024-05-06T04:18:20.499 INFO:tasks.workunit.client.0.smithi163.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "objectstore_numa_unknown_devices": "sda", 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "os": "Linux", 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_data": "td/ceph-helpers/0", 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_objectstore": "bluestore", 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "osdspec_affinity": "", 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "rotational": "1" 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap_clean_epochs": { 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "min_last_epoch_clean": 0, 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "last_epoch_clean": { 2024-05-06T04:18:20.500 INFO:tasks.workunit.client.0.smithi163.stdout: "per_pool": [] 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_epochs": [ 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 5 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap_first_committed": 1, 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap_last_committed": 12, 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "crushmap": { 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: "devices": [ 2024-05-06T04:18:20.501 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "osd.0", 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: "class": "hdd" 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: "types": [ 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 0, 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "osd" 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.502 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 1, 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "host" 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 2, 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "chassis" 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 3, 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "rack" 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.503 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 4, 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "row" 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 5, 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "pdu" 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 6, 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "pod" 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.504 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 7, 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "room" 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 8, 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "datacenter" 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 9, 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "zone" 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.505 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 10, 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "region" 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 11, 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "root" 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "buckets": [ 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -1, 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "default", 2024-05-06T04:18:20.506 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 11, 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "root", 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -3, 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.507 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -2, 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "default~hdd", 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 11, 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "root", 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -4, 2024-05-06T04:18:20.508 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -3, 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "smithi163", 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 1, 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "host", 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:18:20.509 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -4, 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "smithi163~hdd", 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 1, 2024-05-06T04:18:20.510 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "host", 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.511 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "rules": [ 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "rule_id": 0, 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "rule_name": "replicated_rule", 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "type": 1, 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "steps": [ 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "op": "take", 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "item": -1, 2024-05-06T04:18:20.512 INFO:tasks.workunit.client.0.smithi163.stdout: "item_name": "default" 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: "op": "choose_firstn", 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: "num": 0, 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "osd" 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: "op": "emit" 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:18:20.513 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "tunables": { 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_local_tries": 0, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_local_fallback_tries": 0, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_total_tries": 50, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "chooseleaf_descend_once": 1, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "chooseleaf_vary_r": 1, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "chooseleaf_stable": 1, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "msr_descents": 100, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "msr_collision_tries": 100, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "straw_calc_version": 1, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "allowed_bucket_algs": 54, 2024-05-06T04:18:20.514 INFO:tasks.workunit.client.0.smithi163.stdout: "profile": "jewel", 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "optimal_tunables": 1, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "legacy_tunables": 0, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "minimum_required_version": "jewel", 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables": 1, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables2": 1, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v2_rules": 0, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables3": 1, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v3_rules": 0, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v4_buckets": 1, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables5": 1, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v5_rules": 0, 2024-05-06T04:18:20.515 INFO:tasks.workunit.client.0.smithi163.stdout: "has_msr_rules": 0 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_args": {} 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "fsmap": { 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 1, 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "default_fscid": -1, 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "compat": { 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "compat": {}, 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "ro_compat": {}, 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "incompat": { 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_1": "base v0.20", 2024-05-06T04:18:20.516 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_2": "client writeable ranges", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_3": "default file layouts on dirs", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_4": "dir inode in separate object", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_5": "mds uses versioned encoding", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_6": "dirfrag is stored in omap", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_8": "no anchor table", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_9": "file layout v2", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_10": "snaprealm v2", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_11": "minor log segments", 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_12": "quiesce subvolumes" 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.517 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_flags": { 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "enable_multiple": true, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "ever_enabled_multiple": true 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "standbys": [], 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "filesystems": [] 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "mdsmap_first_committed": 1, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "mdsmap_last_committed": 1, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "auth": { 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "first_committed": 1, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "last_committed": 3, 2024-05-06T04:18:20.518 INFO:tasks.workunit.client.0.smithi163.stdout: "num_secrets": 1 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg": 4, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_active": 4, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_unknown": 0, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_osd": 1, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_sum": { 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "stat_sum": { 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes": 19, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects": 1, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_clones": 0, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_copies": 2, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing_on_primary": 0, 2024-05-06T04:18:20.519 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_degraded": 1, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_misplaced": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_unfound": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_dirty": 1, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_whiteouts": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read_kb": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write": 2, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write_kb": 2, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_scrub_errors": 0, 2024-05-06T04:18:20.520 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shallow_scrub_errors": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_deep_scrub_errors": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_recovered": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_recovered": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_keys_recovered": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_omap": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_hit_set_archive": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_hit_set_archive": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_kb": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_kb": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_promote": 0, 2024-05-06T04:18:20.521 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_high": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_low": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_some": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_full": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_pinned": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_legacy_snapsets": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_large_omap_objects": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_manifest": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_bytes": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_keys": 0, 2024-05-06T04:18:20.522 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_repaired": 0 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "store_stats": { 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 0, 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 0 2024-05-06T04:18:20.523 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "log_size": 2, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "ondisk_log_size": 2, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "up": 4, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "acting": 4, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "num_store_stats": 0 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_sum": { 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "up_from": 0, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "seq": 0, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pgs": 4, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "num_osds": 1, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_osds": 1, 2024-05-06T04:18:20.524 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_omap_osds": 1, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "kb": 104857600, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used": 26940, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_data": 108, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_omap": 1, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_meta": 26814, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_avail": 104830660, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "statfs": { 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 107374182400, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 107346595840, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 110592, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 22938, 2024-05-06T04:18:20.525 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 1599, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 27457985 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_peers": [], 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_trim_queue_len": 0, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "num_snap_trimming": 0, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shards_repaired": 0, 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "op_queue_age_hist": { 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "histogram": [], 2024-05-06T04:18:20.526 INFO:tasks.workunit.client.0.smithi163.stdout: "upper_bound": 1 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "perf_stat": { 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ms": 0, 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ms": 0, 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ns": 0, 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ns": 0 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "alerts": [], 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "network_ping_times": [] 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_sum_by_class": { 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "hdd": { 2024-05-06T04:18:20.527 INFO:tasks.workunit.client.0.smithi163.stdout: "up_from": 0, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "seq": 0, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pgs": 4, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "num_osds": 1, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_osds": 1, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_omap_osds": 1, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "kb": 104857600, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used": 26940, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_data": 108, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_omap": 1, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_meta": 26814, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_avail": 104830660, 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "statfs": { 2024-05-06T04:18:20.528 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 107374182400, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 107346595840, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 110592, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 22938, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 1599, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 27457985 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_peers": [], 2024-05-06T04:18:20.529 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_trim_queue_len": 0, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_snap_trimming": 0, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shards_repaired": 0, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "op_queue_age_hist": { 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "histogram": [], 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "upper_bound": 1 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "perf_stat": { 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ms": 0, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ms": 0, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ns": 0, 2024-05-06T04:18:20.530 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ns": 0 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "alerts": [], 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "network_ping_times": [] 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_stats": [ 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "poolid": 1, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg": 4, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "stat_sum": { 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes": 19, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects": 1, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_clones": 0, 2024-05-06T04:18:20.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_copies": 2, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing_on_primary": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_degraded": 1, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_misplaced": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_unfound": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_dirty": 1, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_whiteouts": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read_kb": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write": 2, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write_kb": 2, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_scrub_errors": 0, 2024-05-06T04:18:20.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shallow_scrub_errors": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_deep_scrub_errors": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_recovered": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_recovered": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_keys_recovered": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_omap": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_hit_set_archive": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_hit_set_archive": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_kb": 0, 2024-05-06T04:18:20.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_kb": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_promote": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_high": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_low": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_some": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_full": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_pinned": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_legacy_snapsets": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_large_omap_objects": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_manifest": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_bytes": 0, 2024-05-06T04:18:20.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_keys": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_repaired": 0 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "store_stats": { 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 4096, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 19, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:18:20.535 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 0, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 0 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "log_size": 2, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "ondisk_log_size": 2, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "up": 4, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "acting": 4, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "num_store_stats": 1 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_stats": [ 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:18:20.536 INFO:tasks.workunit.client.0.smithi163.stdout: "seq": 21474836482 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_by_state": [ 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: "state": "active+undersized+degraded", 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num": 1 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: "state": "active+undersized", 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num": 3 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.537 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_by_osd": [ 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "num_primary_pg": 4, 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "num_acting_pg": 4, 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "num_up_not_acting_pg": 0 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "purged_snaps": [ 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:18:20.538 INFO:tasks.workunit.client.0.smithi163.stdout: "purged_snaps": {} 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "servicemap": { 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 1, 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "modified": "2024-05-06T04:17:35.663327+0000", 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "services": {} 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "mgrstat_first_committed": 1, 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "mgrstat_last_committed": 6, 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "logm_first_committed": 1, 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "logm_last_committed": 77, 2024-05-06T04:18:20.539 INFO:tasks.workunit.client.0.smithi163.stdout: "paxos": { 2024-05-06T04:18:20.540 INFO:tasks.workunit.client.0.smithi163.stdout: "first_committed": 1, 2024-05-06T04:18:20.540 INFO:tasks.workunit.client.0.smithi163.stdout: "last_committed": 102, 2024-05-06T04:18:20.540 INFO:tasks.workunit.client.0.smithi163.stdout: "last_pn": 0, 2024-05-06T04:18:20.540 INFO:tasks.workunit.client.0.smithi163.stdout: "accepted_pn": 0 2024-05-06T04:18:20.540 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:18:20.540 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T04:18:20.544 INFO:tasks.workunit.client.0.smithi163.stderr:report 3531182446 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:18:20.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:18:20.546 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:18:20.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:18:20.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:18:20.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:18:20.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:18:20.548 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:18:20.548 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 9c7b9808-cd54-49f9-b4b6-6593fa44ccab 2024-05-06T04:18:20.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c7b9808-cd54-49f9-b4b6-6593fa44ccab 2024-05-06T04:18:20.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9c7b9808-cd54-49f9-b4b6-6593fa44ccab' 2024-05-06T04:18:20.549 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:18:20.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMWjhmAo+XHRAA3NVyBcf4C1FxGd2Oe8C+MQ== 2024-05-06T04:18:20.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMWjhmAo+XHRAA3NVyBcf4C1FxGd2Oe8C+MQ=="}' 2024-05-06T04:18:20.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c7b9808-cd54-49f9-b4b6-6593fa44ccab -i td/ceph-helpers/1/new.json 2024-05-06T04:18:20.966 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:18:20.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:18:20.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMWjhmAo+XHRAA3NVyBcf4C1FxGd2Oe8C+MQ== --osd-uuid 9c7b9808-cd54-49f9-b4b6-6593fa44ccab 2024-05-06T04:18:21.005 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.002+0000 7ff9121a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:18:21.014 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.012+0000 7ff9121a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:18:21.015 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.013+0000 7ff9121a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:18:21.017 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.013+0000 7ff9121a3640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:18:21.017 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.013+0000 7ff9121a3640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:18:21.017 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.013+0000 7ff9121a3640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:18:21.054 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:21.052+0000 7ff9121a3640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:18:27.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:18:27.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:18:27.128 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:18:27.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:18:27.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:18:27.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:18:27.680 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:18:27.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:18:27.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:18:27.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:18:27.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:18:27.706 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:27.703+0000 7f17378fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:18:27.720 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:27.717+0000 7f17378fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:18:27.721 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:27.718+0000 7f17378fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:18:28.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:18:28.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:18:28.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:18:28.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:18:28.143 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:18:28.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:18:28.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:28.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:18:28.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:28.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:28.255 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:28.252+0000 7f17378fb640 -1 Falling back to public interface 2024-05-06T04:18:28.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:29.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:29.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:29.578 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:18:29.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:18:29.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:29.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:30.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:30.741 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:18:30.738+0000 7f17378fb640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:18:31.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:31.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:31.021 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:18:31.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:18:31.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:31.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:31.626 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:32.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:32.627 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:32.627 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:18:32.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:18:32.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:32.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:33.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:34.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:34.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:34.202 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:18:34.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:18:34.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:34.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:34.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:35.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:35.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:35.752 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:18:35.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:18:35.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:35.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:36.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:37.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:37.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:37.189 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:18:37.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:18:37.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:37.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:37.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:38.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:38.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:38.745 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:18:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:18:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:39.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:40.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:40.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:40.150 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:18:40.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:18:40.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:40.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:40.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:41.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:41.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:41.592 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:18:41.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:18:41.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:41.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:41.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:42.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:42.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:42.999 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:18:42.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:18:42.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:42.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:43.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:44.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:44.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:44.675 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:18:44.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:18:44.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:44.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:45.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:46.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:46.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:46.448 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:18:46.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:18:46.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:46.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:47.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:48.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:48.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:48.181 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:18:48.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:18:48.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:48.182 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:48.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:49.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:49.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:49.992 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:18:49.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:18:49.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:49.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:50.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:51.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:51.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:51.905 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:18:51.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:18:51.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:51.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:52.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:53.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:53.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:53.621 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:18:53.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:18:53.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:53.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:54.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:18:55.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:18:55.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:18:55.244 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:18:55.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:18:55.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:18:55.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:18:55.792 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/389441135,v1:127.0.0.1:6811/389441135] [v2:127.0.0.1:6812/389441135,v1:127.0.0.1:6813/389441135] exists,up 9c7b9808-cd54-49f9-b4b6-6593fa44ccab 2024-05-06T04:18:55.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:18:55.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:18:55.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:18:55.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:18:55.794 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:18:55.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:18:55.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:18:55.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:18:55.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:18:55.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:18:55.875 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:18:56.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:18:56.301 INFO:tasks.workunit.client.0.smithi163.stderr:1' 2024-05-06T04:18:56.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:18:56.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:18:56.301 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:18:56.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836491 2024-05-06T04:18:56.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836491 2024-05-06T04:18:56.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491' 2024-05-06T04:18:56.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:18:56.417 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:18:56.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476744 2024-05-06T04:18:56.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476744 2024-05-06T04:18:56.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491 1-68719476744' 2024-05-06T04:18:56.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:18:56.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836491 2024-05-06T04:18:56.532 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:18:56.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:18:56.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836491 2024-05-06T04:18:56.533 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:18:56.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836491 2024-05-06T04:18:56.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-05-06T04:18:56.535 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836491 2024-05-06T04:18:56.535 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:18:56.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836489 -lt 21474836491 2024-05-06T04:18:56.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:18:57.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:18:57.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:18:58.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836491 2024-05-06T04:18:58.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:18:58.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-68719476744 2024-05-06T04:18:58.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:18:58.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:18:58.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-68719476744 2024-05-06T04:18:58.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:18:58.400 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 68719476744 2024-05-06T04:18:58.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476744 2024-05-06T04:18:58.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 68719476744' 2024-05-06T04:18:58.400 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:18:58.830 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476744 -lt 68719476744 2024-05-06T04:18:58.831 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:18:58.831 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:58.831 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:18:59.368 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:18:59.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:18:59.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:18:59.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:18:59.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:19:00.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:19:00.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:19:00.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:19:00.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-05-06T04:19:00.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:19:00.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:19:00.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:19:00.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:19:00.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:19:00.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:19:00.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:19:00.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:19:00.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:19:00.430 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:19:00.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:19:00.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:19:00.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:19:00.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:19:00.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:19:00.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:19:00.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:19:00.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:19:00.433 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:19:00.434 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:19:00.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:19:00.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:19:00.444 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:19:00.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:00.445 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:00.445 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: setup td/ceph-helpers 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:19:00.447 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:19:00.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:19:00.448 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:19:00.448 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:19:00.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:19:00.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:19:00.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:19:00.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:19:00.449 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:19:00.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:19:00.449 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:19:00.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:19:00.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:19:00.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:19:00.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:19:00.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:19:00.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:19:00.451 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:19:00.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:19:00.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:19:00.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:19:00.453 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:19:00.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:00.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:00.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:19:00.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:19:00.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:19:00.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:19:00.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:19:00.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:00.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:00.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1825: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:19:00.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:00.936 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:19:00.937 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-05-06T04:19:01.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:19:01.113 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:19:01.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:19:01.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:19:01.193 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:01.194 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:19:01.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:19:01.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:19:01.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1826: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-05-06T04:19:01.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:19:01.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:19:01.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:19:01.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:19:01.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:19:01.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:19:02.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:19:02.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:19:02.126 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:19:02.127 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:19:02.127 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:02.127 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:02.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:19:02.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:19:02.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-05-06T04:19:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1828: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-05-06T04:19:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:19:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:19:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:19:02.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:19:02.158 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:19:02.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:19:02.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7362b176-5894-40c9-b3ea-3861e1992e8e 2024-05-06T04:19:02.161 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 7362b176-5894-40c9-b3ea-3861e1992e8e 2024-05-06T04:19:02.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7362b176-5894-40c9-b3ea-3861e1992e8e' 2024-05-06T04:19:02.161 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:19:02.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2WjhmLOBTChAA7EtOScMHuNo63Bvv/KV2Kg== 2024-05-06T04:19:02.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2WjhmLOBTChAA7EtOScMHuNo63Bvv/KV2Kg=="}' 2024-05-06T04:19:02.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7362b176-5894-40c9-b3ea-3861e1992e8e -i td/ceph-helpers/0/new.json 2024-05-06T04:19:02.453 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:19:02.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:19:02.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2WjhmLOBTChAA7EtOScMHuNo63Bvv/KV2Kg== --osd-uuid 7362b176-5894-40c9-b3ea-3861e1992e8e 2024-05-06T04:19:02.492 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.489+0000 7f6016e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:02.512 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.510+0000 7f6016e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:02.514 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.511+0000 7f6016e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:02.514 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.511+0000 7f6016e13640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:19:02.514 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.511+0000 7f6016e13640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:19:02.514 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.511+0000 7f6016e13640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:19:02.553 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:02.550+0000 7f6016e13640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:19:08.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:19:08.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:19:08.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:19:08.365 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:19:08.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:19:08.929 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:19:08.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:19:08.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:19:08.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:19:08.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:19:08.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:19:08.955 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:08.952+0000 7f969b9f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:08.966 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:08.963+0000 7f969b9f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:08.967 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:08.965+0000 7f969b9f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:09.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:19:09.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:19:09.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:19:09.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:19:09.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:19:09.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:09.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:19:09.389 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:19:09.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:09.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:09.512 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:09.510+0000 7f969b9f5640 -1 Falling back to public interface 2024-05-06T04:19:09.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:10.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:10.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:10.816 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:19:10.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:19:10.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:10.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:11.240 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:11.851 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:11.848+0000 7f969b9f5640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:19:12.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:12.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:12.242 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:19:12.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:19:12.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:12.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:12.651 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:13.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:13.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:13.653 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:19:13.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:19:13.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:13.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:14.099 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:15.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:15.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:15.101 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:19:15.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:19:15.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:15.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:15.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:16.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:16.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:16.616 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:19:16.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:19:16.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:16.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:17.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:18.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:18.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:18.026 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:19:18.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:19:18.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:18.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:18.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:19.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:19.433 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:19.434 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:19:19.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:19:19.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:19.435 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:19.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:20.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:20.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:20.926 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:19:20.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:19:20.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:20.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:21.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:22.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:22.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:22.340 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:19:22.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:19:22.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:22.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:22.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:23.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:23.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:23.753 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:19:23.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:19:23.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:23.753 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:24.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:25.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:25.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:25.612 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:19:25.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:19:25.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:25.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:26.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:27.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:27.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:27.357 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:19:27.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:19:27.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:27.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:28.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:29.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:29.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:29.192 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:19:29.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:19:29.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:29.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:29.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:30.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:30.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:30.944 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:19:30.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:19:30.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:30.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:31.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:32.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:32.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:32.645 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:19:32.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:19:32.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:32.646 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:33.441 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:34.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:34.442 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:34.443 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:19:34.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:19:34.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:34.443 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:19:34.911 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/821657220,v1:127.0.0.1:6803/821657220] [v2:127.0.0.1:6804/821657220,v1:127.0.0.1:6805/821657220] exists,up 7362b176-5894-40c9-b3ea-3861e1992e8e 2024-05-06T04:19:34.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:19:34.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:19:34.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:19:34.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:19:34.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c4fab62-d523-43d3-ae3c-c59de0a44b2a 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7c4fab62-d523-43d3-ae3c-c59de0a44b2a' 2024-05-06T04:19:34.918 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 7c4fab62-d523-43d3-ae3c-c59de0a44b2a 2024-05-06T04:19:34.919 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:19:34.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWWjhm34VrNxAATxkpOfNGixTDi3cp9XhPiw== 2024-05-06T04:19:34.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWWjhm34VrNxAATxkpOfNGixTDi3cp9XhPiw=="}' 2024-05-06T04:19:34.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c4fab62-d523-43d3-ae3c-c59de0a44b2a -i td/ceph-helpers/1/new.json 2024-05-06T04:19:35.414 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:19:35.425 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:19:35.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWWjhm34VrNxAATxkpOfNGixTDi3cp9XhPiw== --osd-uuid 7c4fab62-d523-43d3-ae3c-c59de0a44b2a 2024-05-06T04:19:35.453 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.450+0000 7f352e415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:35.463 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.460+0000 7f352e415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:35.463 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.461+0000 7f352e415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:35.464 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.461+0000 7f352e415640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:19:35.464 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.461+0000 7f352e415640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:19:35.464 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.461+0000 7f352e415640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:19:35.502 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:35.499+0000 7f352e415640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:19:41.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:19:41.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:19:41.515 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:19:41.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:19:41.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:19:42.065 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:19:42.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:19:42.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:19:42.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:19:42.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:19:42.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:19:42.090 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:42.087+0000 7f82acd08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:42.107 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:42.104+0000 7f82acd08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:42.108 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:42.105+0000 7f82acd08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:19:42.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:19:42.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:19:42.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:19:42.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:19:42.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:19:42.495 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:19:42.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:42.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:19:42.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:42.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:42.639 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:42.637+0000 7f82acd08640 -1 Falling back to public interface 2024-05-06T04:19:42.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:43.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:43.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:43.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:19:43.931 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:19:43.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:43.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:44.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:45.050 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:19:45.047+0000 7f82acd08640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:19:45.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:45.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:45.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:19:45.339 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:19:45.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:45.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:45.867 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:46.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:46.869 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:46.869 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:19:46.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:19:46.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:46.870 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:47.299 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:48.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:48.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:48.301 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:19:48.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:19:48.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:48.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:48.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:49.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:49.704 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:19:49.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:49.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:19:49.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:49.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:50.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:51.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:51.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:51.113 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:19:51.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:19:51.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:51.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:51.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:52.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:52.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:52.653 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:19:52.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:19:52.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:52.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:53.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:54.058 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:54.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:54.059 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:19:54.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:19:54.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:54.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:54.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:55.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:55.470 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:55.471 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:19:55.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:19:55.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:55.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:55.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:56.903 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:19:56.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:19:56.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:56.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:57.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:19:58.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:19:58.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:19:58.456 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:19:58.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:19:58.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:19:58.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:19:59.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:00.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:00.158 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:00.158 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:20:00.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:20:00.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:00.159 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:00.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:01.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:01.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:01.810 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:20:01.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:20:01.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:01.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:02.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:03.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:03.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:03.504 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:20:03.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:20:03.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:03.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:04.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:05.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:05.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:05.327 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:20:05.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:20:05.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:05.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:06.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:07.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:07.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:07.103 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:20:07.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:20:07.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:07.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:08.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:09.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:09.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:09.022 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:20:09.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:20:09.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:09.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:09.444 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/639357956,v1:127.0.0.1:6811/639357956] [v2:127.0.0.1:6812/639357956,v1:127.0.0.1:6813/639357956] exists,up 7c4fab62-d523-43d3-ae3c-c59de0a44b2a 2024-05-06T04:20:09.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:20:09.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:20:09.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:20:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:20:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f5c4fd4-77c7-4df3-937a-43ab84bf5585 2024-05-06T04:20:09.451 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 2f5c4fd4-77c7-4df3-937a-43ab84bf5585 2024-05-06T04:20:09.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2f5c4fd4-77c7-4df3-937a-43ab84bf5585' 2024-05-06T04:20:09.452 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:20:09.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5WjhmySSvGxAAithHs+lQPOH/2NmBrviYdw== 2024-05-06T04:20:09.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5WjhmySSvGxAAithHs+lQPOH/2NmBrviYdw=="}' 2024-05-06T04:20:09.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f5c4fd4-77c7-4df3-937a-43ab84bf5585 -i td/ceph-helpers/2/new.json 2024-05-06T04:20:10.038 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:20:10.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-05-06T04:20:10.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5WjhmySSvGxAAithHs+lQPOH/2NmBrviYdw== --osd-uuid 2f5c4fd4-77c7-4df3-937a-43ab84bf5585 2024-05-06T04:20:10.079 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.074+0000 7f1a6b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:10.079 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.076+0000 7f1a6b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:10.080 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.077+0000 7f1a6b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:10.081 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.077+0000 7f1a6b211640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:20:10.081 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.077+0000 7f1a6b211640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:20:10.081 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.077+0000 7f1a6b211640 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-05-06T04:20:10.142 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:10.139+0000 7f1a6b211640 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-05-06T04:20:16.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-05-06T04:20:16.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:20:16.455 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:20:16.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:20:16.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:20:17.004 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:20:17.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:20:17.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:20:17.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:20:17.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:20:17.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:20:17.031 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:17.028+0000 7fbea1a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:17.057 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:17.054+0000 7fbea1a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:17.058 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:17.055+0000 7fbea1a2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:17.455 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:20:17.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:20:17.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:17.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:17.590 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:17.587+0000 7fbea1a2e640 -1 Falling back to public interface 2024-05-06T04:20:17.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:18.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:18.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:18.890 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:20:18.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:20:18.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:18.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:19.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:19.942 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:19.939+0000 7fbea1a2e640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:20:20.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:20.325 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:20:20.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:20.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:20:20.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:20.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:20.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:21.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:21.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:21.772 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:20:21.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:20:21.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:21.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:22.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:23.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:23.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:23.192 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:20:23.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:20:23.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:23.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:23.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:24.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:24.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:24.616 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:20:24.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:20:24.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:24.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:25.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:26.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:26.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:26.084 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:20:26.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:20:26.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:26.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:26.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:27.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:27.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:27.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:20:27.497 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:20:27.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:27.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:27.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:28.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:28.968 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:20:28.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:28.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:20:28.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:28.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:29.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:30.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:30.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:30.475 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:20:30.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:20:30.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:30.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:30.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:31.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:31.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:31.884 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:20:31.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:20:31.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:31.885 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:32.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:33.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:33.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:33.386 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:20:33.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:20:33.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:33.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:34.267 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:35.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:35.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:35.270 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:20:35.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:20:35.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:35.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:36.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:37.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:37.026 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:20:37.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:37.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:20:37.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:37.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:38.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:39.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:39.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:39.120 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:20:39.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:20:39.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:39.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:39.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:40.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:40.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:40.901 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:20:40.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:20:40.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:40.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:41.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:42.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:42.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:42.816 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:20:42.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:20:42.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:42.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:43.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:44.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:44.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:44.405 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:20:44.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:20:44.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:44.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:44.806 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3388786124,v1:127.0.0.1:6819/3388786124] [v2:127.0.0.1:6820/3388786124,v1:127.0.0.1:6821/3388786124] exists,up 2f5c4fd4-77c7-4df3-937a-43ab84bf5585 2024-05-06T04:20:44.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:20:44.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:20:44.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:20:44.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-05-06T04:20:44.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:20:44.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:20:44.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:20:44.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:20:44.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:20:45.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:20:45.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: test_wait_for_health_ok: ceph osd down 0 2024-05-06T04:20:45.574 INFO:tasks.workunit.client.0.smithi163.stderr:osd.0 is already down. 2024-05-06T04:20:45.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: TIMEOUT=1 2024-05-06T04:20:45.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_wait_for_health_ok: wait_for_health_ok 2024-05-06T04:20:45.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-05-06T04:20:45.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-05-06T04:20:45.586 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 1 .1 2024-05-06T04:20:45.587 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:20:45.587 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:20:45.587 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:20:45.587 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:20:45.587 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:20:45.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-05-06T04:20:45.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-05-06T04:20:45.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-05-06T04:20:45.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-05-06T04:20:45.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-05-06T04:20:46.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 0 >= 4 )) 2024-05-06T04:20:46.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.1 2024-05-06T04:20:46.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-05-06T04:20:46.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-05-06T04:20:46.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-05-06T04:20:46.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 1 >= 4 )) 2024-05-06T04:20:46.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.2 2024-05-06T04:20:47.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-05-06T04:20:47.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-05-06T04:20:47.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-05-06T04:20:47.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 2 >= 4 )) 2024-05-06T04:20:47.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.4 2024-05-06T04:20:47.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-05-06T04:20:47.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-05-06T04:20:47.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-05-06T04:20:48.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 3 >= 4 )) 2024-05-06T04:20:48.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: wait_for_health: sleep 0.3 2024-05-06T04:20:48.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health: loop+=1 2024-05-06T04:20:48.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-05-06T04:20:48.801 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-05-06T04:20:49.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health: (( 4 >= 4 )) 2024-05-06T04:20:49.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1803: wait_for_health: ceph health detail 2024-05-06T04:20:49.888 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-05-06T04:20:49.889 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] OSD_DOWN: 3 osds down 2024-05-06T04:20:49.889 INFO:tasks.workunit.client.0.smithi163.stdout: osd.0 (root=default,host=smithi163) is down 2024-05-06T04:20:49.889 INFO:tasks.workunit.client.0.smithi163.stdout: osd.1 (root=default,host=smithi163) is down 2024-05-06T04:20:49.889 INFO:tasks.workunit.client.0.smithi163.stdout: osd.2 (root=default,host=smithi163) is down 2024-05-06T04:20:49.889 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-05-06T04:20:49.889 INFO:tasks.workunit.client.0.smithi163.stdout: host smithi163 (root=default) (3 osds) is down 2024-05-06T04:20:49.890 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-05-06T04:20:49.890 INFO:tasks.workunit.client.0.smithi163.stdout: root default (3 osds) is down 2024-05-06T04:20:49.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1804: wait_for_health: return 1 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: return 1 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:20:49.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:20:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:20:49.906 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:20:49.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T04:20:49.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:20:49.908 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-05-06T04:20:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T04:20:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:20:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:20:49.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:20:49.928 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:49.926+0000 7f26dec13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:49.930 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:49.927+0000 7f26dec13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:49.931 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:49.929+0000 7f26dec13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:50.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T04:20:50.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:20:50.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:20:50.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:20:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:20:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:20:50.368 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:20:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:50.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:20:50.453 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:50.450+0000 7f26dec13640 -1 Falling back to public interface 2024-05-06T04:20:50.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:51.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:51.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:51.793 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:20:51.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:20:51.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:51.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:20:52.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:53.162 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:53.159+0000 7f26dec13640 -1 osd.0 14 log_to_monitors true 2024-05-06T04:20:53.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:53.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:53.230 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:20:53.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:20:53.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:53.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:20:53.658 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,14) [v2:127.0.0.1:6802/330741827,v1:127.0.0.1:6803/330741827] [v2:127.0.0.1:6804/330741827,v1:127.0.0.1:6805/330741827] exists,up 7362b176-5894-40c9-b3ea-3861e1992e8e 2024-05-06T04:20:53.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:20:53.661 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:20:53.662 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:20:53.663 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:20:53.664 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:20:53.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-06T04:20:53.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:20:53.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-05-06T04:20:53.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-06T04:20:53.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:20:53.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:20:53.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:20:53.686 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:53.683+0000 7f1ab8385640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:53.716 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:53.713+0000 7f1ab8385640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:53.716 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:53.714+0000 7f1ab8385640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:54.095 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:20:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:20:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:54.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:54.233 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:54.230+0000 7f1ab8385640 -1 Falling back to public interface 2024-05-06T04:20:54.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:55.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:55.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:55.489 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:20:55.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:20:55.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:55.490 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:55.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:56.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:56.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:56.900 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:20:56.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:20:56.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:56.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:20:56.963 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:56.960+0000 7f1ab8385640 -1 osd.1 16 log_to_monitors true 2024-05-06T04:20:57.371 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/2482484475,v1:127.0.0.1:6811/2482484475] [v2:127.0.0.1:6812/2482484475,v1:127.0.0.1:6813/2482484475] exists,up 7c4fab62-d523-43d3-ae3c-c59de0a44b2a 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:20:57.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:57.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-05-06T04:20:57.377 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:20:57.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-06T04:20:57.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:20:57.378 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-05-06T04:20:57.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-06T04:20:57.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:20:57.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:20:57.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:20:57.400 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:57.397+0000 7f82c33da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:57.426 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:57.423+0000 7f82c33da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:57.427 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:57.424+0000 7f82c33da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:20:57.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-06T04:20:57.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:20:57.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:20:57.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:20:57.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:20:57.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:57.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:20:57.806 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:20:57.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:57.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:57.949 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:20:57.946+0000 7f82c33da640 -1 Falling back to public interface 2024-05-06T04:20:58.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:20:59.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:20:59.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:20:59.207 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:20:59.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:20:59.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:20:59.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:20:59.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:00.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:00.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:00.641 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:21:00.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:21:00.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:00.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:21:00.748 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:00.746+0000 7f82c33da640 -1 osd.2 15 log_to_monitors true 2024-05-06T04:21:01.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:02.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:02.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:02.080 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:21:02.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:21:02.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:02.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:21:02.523 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6818/2084590245,v1:127.0.0.1:6819/2084590245] [v2:127.0.0.1:6820/2084590245,v1:127.0.0.1:6821/2084590245] exists,up 2f5c4fd4-77c7-4df3-937a-43ab84bf5585 2024-05-06T04:21:02.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:21:02.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: test_wait_for_health_ok: wait_for_health_ok 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: wait_for_health_ok: wait_for_health HEALTH_OK 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: wait_for_health: local grepstr=HEALTH_OK 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: get_timeout_delays 300 .1 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:21:02.525 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:21:02.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-05-06T04:21:02.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: wait_for_health: local -a delays 2024-05-06T04:21:02.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: wait_for_health: local -i loop=0 2024-05-06T04:21:02.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: ceph health detail 2024-05-06T04:21:02.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1801: wait_for_health: grep HEALTH_OK 2024-05-06T04:21:03.224 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_OK 2024-05-06T04:21:03.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: test_wait_for_health_ok: teardown td/ceph-helpers 2024-05-06T04:21:03.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:21:03.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:21:03.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:21:03.225 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:21:03.226 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:21:03.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:21:03.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:21:03.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:21:03.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:21:03.333 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:21:03.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:21:03.334 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:21:03.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:21:03.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:21:03.336 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:21:03.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:21:03.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:21:03.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:21:03.337 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:21:03.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:21:03.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:21:03.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:21:03.350 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:21:03.350 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:03.350 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:03.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_osd td/ceph-helpers 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:21:03.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:21:03.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:21:03.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:21:03.353 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:21:03.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:21:03.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:21:03.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:21:03.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:21:03.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:21:03.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:21:03.354 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:21:03.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:21:03.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:21:03.355 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:21:03.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:21:03.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:21:03.356 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:21:03.356 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:21:03.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:21:03.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:21:03.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:21:03.359 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:21:03.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:03.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:03.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:21:03.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:21:03.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:21:03.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:21:03.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:21:03.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:03.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:03.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:21:03.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:21:03.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:21:03.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:21:03.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:21:03.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:21:03.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:21:03.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:21:03.836 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:21:03.836 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:03.836 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:03.837 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:21:03.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:21:04.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:21:04.038 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:21:04.038 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:21:04.038 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:21:04.038 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:21:04.039 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:21:04.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:21:04.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:21:04.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:21:04.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:21:05.060 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:21:05.060 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:21:05.061 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:21:05.061 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:21:05.061 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:05.061 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:05.061 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:21:05.061 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:21:05.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:21:05.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:21:05.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:21:05.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a81f36ad-c112-49c0-93d1-6845ca29eb77 2024-05-06T04:21:05.094 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 a81f36ad-c112-49c0-93d1-6845ca29eb77 2024-05-06T04:21:05.095 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a81f36ad-c112-49c0-93d1-6845ca29eb77' 2024-05-06T04:21:05.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:21:05.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxWjhmhslmBhAAKGne9n+ROMmh/PlWqV8fzg== 2024-05-06T04:21:05.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxWjhmhslmBhAAKGne9n+ROMmh/PlWqV8fzg=="}' 2024-05-06T04:21:05.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a81f36ad-c112-49c0-93d1-6845ca29eb77 -i td/ceph-helpers/0/new.json 2024-05-06T04:21:05.403 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:21:05.413 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:21:05.414 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxWjhmhslmBhAAKGne9n+ROMmh/PlWqV8fzg== --osd-uuid a81f36ad-c112-49c0-93d1-6845ca29eb77 2024-05-06T04:21:05.442 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.439+0000 7ffad5ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:05.471 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.469+0000 7ffad5ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:05.472 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.470+0000 7ffad5ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:05.473 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.470+0000 7ffad5ba3640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:21:05.473 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.470+0000 7ffad5ba3640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:21:05.473 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.470+0000 7ffad5ba3640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:21:05.512 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:05.509+0000 7ffad5ba3640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:21:11.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:21:11.276 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:21:11.277 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:21:11.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:21:11.277 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:21:11.846 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:21:11.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:21:11.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:21:11.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:21:11.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:21:11.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:21:11.872 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:11.870+0000 7fb430e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:11.892 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:11.889+0000 7fb430e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:11.893 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:11.890+0000 7fb430e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:12.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:21:12.316 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:21:12.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:12.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:12.428 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:12.426+0000 7fb430e11640 -1 Falling back to public interface 2024-05-06T04:21:12.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:13.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:13.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:13.742 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:21:13.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:21:13.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:13.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:14.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:14.786 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:14.783+0000 7fb430e11640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:21:15.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:15.167 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:15.167 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:21:15.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:21:15.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:15.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:15.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:16.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:16.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:16.573 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:21:16.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:21:16.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:16.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:16.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:17.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:17.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:17.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:21:17.993 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:21:17.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:17.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:18.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:19.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:19.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:19.400 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:21:19.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:21:19.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:19.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:19.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:20.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:20.946 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:20.946 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:21:20.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:21:20.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:20.947 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:21.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:22.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:22.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:22.354 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:21:22.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:21:22.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:22.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:22.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:23.781 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:23.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:23.782 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:21:23.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:21:23.782 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:23.783 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:24.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:25.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:25.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:25.186 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:21:25.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:21:25.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:25.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:25.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:26.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:26.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:26.718 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:21:26.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:21:26.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:26.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:27.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:28.339 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:28.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:28.340 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:21:28.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:21:28.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:28.341 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:29.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:30.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:30.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:30.035 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:21:30.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:21:30.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:30.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:30.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:31.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:31.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:31.771 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:21:31.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:21:31.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:31.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:32.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:33.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:33.534 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:33.535 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:21:33.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:21:33.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:33.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:34.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:35.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:35.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:35.353 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:21:35.353 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:21:35.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:35.354 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:35.752 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:36.754 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:21:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:21:36.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:36.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:37.510 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:38.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:38.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:38.512 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:21:38.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:21:38.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:38.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:21:38.921 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1385030575,v1:127.0.0.1:6803/1385030575] [v2:127.0.0.1:6804/1385030575,v1:127.0.0.1:6805/1385030575] exists,up a81f36ad-c112-49c0-93d1-6845ca29eb77 2024-05-06T04:21:38.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:21:38.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:21:38.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:21:38.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:21:38.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb784b45-6908-4f63-871f-441aea6d354a 2024-05-06T04:21:38.929 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 cb784b45-6908-4f63-871f-441aea6d354a 2024-05-06T04:21:38.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cb784b45-6908-4f63-871f-441aea6d354a' 2024-05-06T04:21:38.929 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:21:38.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSWjhmP8EROBAA5uG9/1ZDVVR5AYg2mWwT/g== 2024-05-06T04:21:38.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSWjhmP8EROBAA5uG9/1ZDVVR5AYg2mWwT/g=="}' 2024-05-06T04:21:38.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb784b45-6908-4f63-871f-441aea6d354a -i td/ceph-helpers/1/new.json 2024-05-06T04:21:39.420 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:21:39.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:21:39.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSWjhmP8EROBAA5uG9/1ZDVVR5AYg2mWwT/g== --osd-uuid cb784b45-6908-4f63-871f-441aea6d354a 2024-05-06T04:21:39.460 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.457+0000 7f60d5e63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:39.468 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.466+0000 7f60d5e63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:39.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.467+0000 7f60d5e63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:39.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.467+0000 7f60d5e63640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:21:39.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.467+0000 7f60d5e63640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:21:39.470 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.467+0000 7f60d5e63640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:21:39.508 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:39.505+0000 7f60d5e63640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:21:45.531 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:21:45.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:21:45.532 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:21:45.532 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:21:45.533 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:21:46.087 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:21:46.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:21:46.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:21:46.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:21:46.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:21:46.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:21:46.114 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:46.111+0000 7fc04a00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:46.123 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:46.119+0000 7fc04a00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:46.124 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:46.121+0000 7fc04a00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:21:46.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:21:46.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:21:46.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:21:46.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:21:46.537 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:21:46.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:21:46.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:46.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:21:46.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:46.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:46.656 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:46.654+0000 7fc04a00f640 -1 Falling back to public interface 2024-05-06T04:21:46.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:47.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:47.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:47.974 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:21:47.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:21:47.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:47.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:48.376 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:49.017 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:21:49.014+0000 7fc04a00f640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:21:49.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:49.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:49.378 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:21:49.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:21:49.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:49.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:49.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:50.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:50.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:50.841 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:21:50.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:21:50.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:50.841 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:51.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:52.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:52.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:52.259 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:21:52.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:21:52.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:52.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:52.706 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:53.707 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:53.707 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:53.707 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:21:53.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:21:53.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:53.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:54.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:55.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:55.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:55.120 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:21:55.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:21:55.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:55.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:55.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:56.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:56.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:56.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:21:56.544 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:21:56.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:56.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:56.964 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:57.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:57.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:57.966 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:21:57.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:21:57.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:57.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:58.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:21:59.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:21:59.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:21:59.457 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:21:59.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:21:59.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:21:59.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:21:59.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:00.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:00.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:00.987 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:22:00.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:22:00.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:00.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:01.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:02.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:02.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:02.597 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:22:02.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:22:02.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:02.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:03.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:04.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:04.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:04.406 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:22:04.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:22:04.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:04.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:05.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:06.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:06.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:06.200 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:22:06.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:22:06.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:06.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:07.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:08.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:08.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:08.085 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:22:08.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:22:08.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:08.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:09.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:10.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:10.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:10.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:22:10.239 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:22:10.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:10.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:11.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:12.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:12.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:12.303 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:22:12.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:22:12.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:12.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:12.784 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/691829267,v1:127.0.0.1:6811/691829267] [v2:127.0.0.1:6812/691829267,v1:127.0.0.1:6813/691829267] exists,up cb784b45-6908-4f63-871f-441aea6d354a 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:12.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:22:12.787 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:22:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:12.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:13.228 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1385030575,v1:127.0.0.1:6803/1385030575] [v2:127.0.0.1:6804/1385030575,v1:127.0.0.1:6805/1385030575] exists,up a81f36ad-c112-49c0-93d1-6845ca29eb77 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:13.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:22:13.231 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:22:13.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:13.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:22:13.715 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/691829267,v1:127.0.0.1:6811/691829267] [v2:127.0.0.1:6812/691829267,v1:127.0.0.1:6813/691829267] exists,up cb784b45-6908-4f63-871f-441aea6d354a 2024-05-06T04:22:13.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:22:13.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:22:13.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:22:13.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-05-06T04:22:13.716 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:22:13.716 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:22:13.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:22:13.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:22:13.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:14.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:22:14.024 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:22:14.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:14.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-05-06T04:22:14.435 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/1385030575,v1:127.0.0.1:6803/1385030575] [v2:127.0.0.1:6804/1385030575,v1:127.0.0.1:6805/1385030575] exists a81f36ad-c112-49c0-93d1-6845ca29eb77 2024-05-06T04:22:14.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:22:14.436 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-05-06T04:22:14.437 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:22:14.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:22:14.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:14.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:14.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:22:15.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:22:15.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:22:16.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:22:16.004 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:22:16.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:22:16.005 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:22:16.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:22:16.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:22:16.007 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:22:16.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:22:16.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:22:16.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:22:16.008 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:22:16.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:22:16.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:22:16.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:22:16.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:22:16.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:22:16.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:22:16.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:22:16.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:22:16.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_peered td/ceph-helpers 2024-05-06T04:22:16.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-05-06T04:22:16.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:22:16.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:22:16.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:22:16.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:22:16.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:22:16.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:22:16.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:22:16.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:22:16.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:22:16.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:22:16.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:22:16.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:22:16.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:22:16.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:22:16.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:22:16.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:22:16.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:22:16.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:22:16.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:22:16.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:22:16.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:22:16.032 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.032 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:22:16.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:22:16.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:22:16.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:22:16.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:22:16.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-05-06T04:22:16.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:22:16.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:22:16.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:22:16.499 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:22:16.499 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.500 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.500 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:22:16.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-05-06T04:22:16.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:22:16.701 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:22:16.702 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:22:16.702 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.702 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.703 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:22:16.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:22:16.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:22:16.787 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:22:16.788 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:22:16.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-05-06T04:22:16.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:22:16.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:22:16.871 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:22:16.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:22:16.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9b56e9e-199b-4395-8b4e-489e042a96dd 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e9b56e9e-199b-4395-8b4e-489e042a96dd' 2024-05-06T04:22:16.874 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 e9b56e9e-199b-4395-8b4e-489e042a96dd 2024-05-06T04:22:16.875 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:22:16.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4WjhmMnzfNBAANr7vvF9El91OIYQCBJ3xNA== 2024-05-06T04:22:16.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4WjhmMnzfNBAANr7vvF9El91OIYQCBJ3xNA=="}' 2024-05-06T04:22:16.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9b56e9e-199b-4395-8b4e-489e042a96dd -i td/ceph-helpers/0/new.json 2024-05-06T04:22:17.739 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:22:17.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:22:17.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4WjhmMnzfNBAANr7vvF9El91OIYQCBJ3xNA== --osd-uuid e9b56e9e-199b-4395-8b4e-489e042a96dd 2024-05-06T04:22:17.778 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.774+0000 7f9b85eff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:22:17.807 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.804+0000 7f9b85eff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:22:17.808 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.805+0000 7f9b85eff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:22:17.809 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.806+0000 7f9b85eff640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:22:17.809 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.806+0000 7f9b85eff640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:22:17.809 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.806+0000 7f9b85eff640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:22:17.848 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:17.845+0000 7f9b85eff640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:22:23.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:22:23.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:22:23.076 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:22:23.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:22:23.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:22:23.283 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:22:23.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:22:23.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:22:23.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:22:23.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:22:23.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:22:23.310 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:23.307+0000 7f06e95d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:22:23.317 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:23.314+0000 7f06e95d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:22:23.318 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:23.315+0000 7f06e95d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:22:23.494 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:22:23.495 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:22:23.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:23.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:23.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:23.846 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:23.844+0000 7f06e95d6640 -1 Falling back to public interface 2024-05-06T04:22:24.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:24.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:24.705 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:22:24.706 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:22:24.706 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:24.706 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:24.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:25.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:25.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:25.905 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:22:25.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:22:25.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:25.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:26.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:26.234 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:22:26.232+0000 7f06e95d6640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:22:27.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:27.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:27.103 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:22:27.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:22:27.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:27.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:27.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:28.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:28.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:28.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:22:28.385 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:22:28.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:28.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:28.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:29.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:29.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:29.587 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:22:29.587 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:22:29.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:29.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:29.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:30.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:30.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:30.788 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:22:30.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:22:30.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:30.788 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:30.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:31.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:31.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:31.988 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:22:31.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:22:31.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:31.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:32.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:33.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:33.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:33.189 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:22:33.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:22:33.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:33.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:33.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:34.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:34.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:34.389 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:22:34.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:22:34.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:34.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:34.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:35.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:35.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:35.591 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:22:35.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:22:35.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:35.592 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:35.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:36.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:36.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:36.792 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:22:36.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:22:36.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:36.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:36.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:37.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:37.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:37.995 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:22:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:22:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:37.996 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:38.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:39.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:39.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:39.197 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:22:39.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:22:39.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:39.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:39.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:40.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:40.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:40.399 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:22:40.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:22:40.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:40.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:40.599 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:41.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:41.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:41.601 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:22:41.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:22:41.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:41.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:41.800 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:42.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:42.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:42.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:22:42.802 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:22:42.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:42.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:42.997 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:43.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:43.998 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:43.998 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:22:43.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:22:43.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:43.999 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:44.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:45.200 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:45.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:45.201 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:22:45.201 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:22:45.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:45.202 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:45.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:46.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:46.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:46.407 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:22:46.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:22:46.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:46.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:46.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:22:47.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:22:47.609 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:22:47.609 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:22:47.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:22:47.610 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:22:47.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:22:47.809 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/275213922,v1:127.0.0.1:6801/275213922] [v2:127.0.0.1:6802/275213922,v1:127.0.0.1:6803/275213922] exists,up e9b56e9e-199b-4395-8b4e-489e042a96dd 2024-05-06T04:22:47.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:22:47.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:22:47.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:22:47.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-05-06T04:22:47.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:22:47.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:22:47.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:22:47.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:22:47.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:22:47.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:22:48.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:22:48.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:22:48.066 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:22:48.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:22:48.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:22:48.067 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:22:48.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:22:48.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:22:48.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:22:48.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-05-06T04:22:48.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:22:48.324 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:22:48.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:22:48.334 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:22:48.683 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:22:48.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:22:49.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:22:50.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-05-06T04:22:50.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-05-06T04:22:50.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:22:50.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:22:50.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:22:50.613 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-05-06T04:22:50.614 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:22:50.614 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:22:50.614 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:22:50.614 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:22:50.614 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:22:50.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-05-06T04:22:50.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:22:50.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:22:50.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:22:50.639 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:22:50.639 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:22:50.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:22:50.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:22:50.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:22:50.862 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:22:50.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=0 2024-05-06T04:22:50.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 0 2024-05-06T04:22:50.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-0' 2024-05-06T04:22:50.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:22:50.991 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-0 2024-05-06T04:22:50.991 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:22:50.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:22:50.992 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-0 2024-05-06T04:22:50.993 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:22:50.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=0 2024-05-06T04:22:50.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-05-06T04:22:50.994 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 0 2024-05-06T04:22:50.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:22:51.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 0 -lt 0 2024-05-06T04:22:51.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:22:51.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:51.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:22:51.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-05-06T04:22:51.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-05-06T04:22:52.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:22:52.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:52.401 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:22:53.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:22:53.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:22:53.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:22:53.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:53.428 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:22:53.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:22:53.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-05-06T04:22:53.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-06T04:22:53.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-05-06T04:22:53.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:22:53.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-06T04:22:54.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:22:54.064 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:22:54.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:22:54.491 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:22:54.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:54.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:22:55.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:22:55.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:22:55.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:22:55.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:22:55.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:22:55.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:22:55.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:22:55.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:22:55.024 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:22:55.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:22:55.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:22:55.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-05-06T04:22:55.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:22:55.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-05-06T04:22:55.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:22:55.791 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:22:55.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:22:55.791 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:22:55.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:22:55.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:22:55.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:22:55.792 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:22:56.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:22:56.176 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:22:56.176 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:56.176 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:22:56.709 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:22:57.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:22:57.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:22:57.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-05-06T04:22:57.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:22:57.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-05-06T04:22:57.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:22:57.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:22:57.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:22:57.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:22:57.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:22:57.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:22:57.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:22:57.677 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:22:58.059 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:22:58.059 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:22:58.060 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:58.060 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:22:58.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:22:58.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:22:58.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:22:58.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:22:58.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:22:58.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:22:58.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:22:58.596 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:22:58.596 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:22:59.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:22:59.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:22:59.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-05-06T04:22:59.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-06T04:22:59.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-05-06T04:22:59.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-06T04:22:59.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:22:59.430 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:22:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:22:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:22:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:22:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:22:59.431 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:22:59.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-05-06T04:22:59.812 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:22:59.812 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:22:59.812 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-06T04:23:00.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-06T04:23:00.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-06T04:23:00.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-06T04:23:00.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-05-06T04:23:00.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout:{ 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_fingerprint": "f4e858df-fd3f-44b4-b55e-7eeb9fa93d10", 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "version": "19.0.0-3516-ge08cbbf6", 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "commit": "e08cbbf6fd79a8dd2949f340bce25e28ea3db004", 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "timestamp": "2024-05-06T04:23:01.483478+0000", 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "tag": "", 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "health": { 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "status": "HEALTH_WARN", 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "checks": { 2024-05-06T04:23:01.486 INFO:tasks.workunit.client.0.smithi163.stdout: "PG_DEGRADED": { 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "severity": "HEALTH_WARN", 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "summary": { 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "message": "Degraded data redundancy: 1/2 objects degraded (50.000%), 1 pg degraded", 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "count": 1 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "detail": [ 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "muted": false 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.487 INFO:tasks.workunit.client.0.smithi163.stdout: "TOO_FEW_OSDS": { 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "severity": "HEALTH_WARN", 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "summary": { 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "count": 2 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "detail": [], 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "muted": false 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "mutes": [] 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "monmap_first_committed": 1, 2024-05-06T04:23:01.488 INFO:tasks.workunit.client.0.smithi163.stdout: "monmap_last_committed": 1, 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "monmap": { 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 1, 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "fsid": "fc6d983f-746c-4b93-98c2-58f795783e09", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "modified": "2024-05-06T04:22:16.053798Z", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "created": "2024-05-06T04:22:16.053798Z", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "min_mon_release": 19, 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "min_mon_release_name": "squid", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "election_strategy": 1, 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "disallowed_leaders: ": "", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode": false, 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "tiebreaker_mon": "", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "removed_ranks: ": "", 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "features": { 2024-05-06T04:23:01.489 INFO:tasks.workunit.client.0.smithi163.stdout: "persistent": [ 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "kraken", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "luminous", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "mimic", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap-prune", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "nautilus", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "octopus", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "pacific", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "elector-pinging", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "quincy", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "reef", 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "squid" 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.490 INFO:tasks.workunit.client.0.smithi163.stdout: "optional": [] 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "mons": [ 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "rank": 0, 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "a", 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addrs": { 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:7109", 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 0 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.491 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:7109/0", 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addr": "127.0.0.1:7109/0", 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: "priority": 0, 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 0, 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_location": "{}" 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: "quorum": [ 2024-05-06T04:23:01.492 INFO:tasks.workunit.client.0.smithi163.stdout: 0 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap": { 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 12, 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "fsid": "fc6d983f-746c-4b93-98c2-58f795783e09", 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "created": "2024-05-06T04:22:16.944375+0000", 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "modified": "2024-05-06T04:22:50.290307+0000", 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "last_up_change": "2024-05-06T04:22:47.171950+0000", 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "last_in_change": "2024-05-06T04:22:17.643714+0000", 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "flags_num": 5799936, 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "flags_set": [ 2024-05-06T04:23:01.493 INFO:tasks.workunit.client.0.smithi163.stdout: "pglog_hardlimit", 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: "purged_snapdirs", 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: "recovery_deletes", 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: "sortbitwise" 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_version": 3, 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: "full_ratio": 0.99000000953674316, 2024-05-06T04:23:01.494 INFO:tasks.workunit.client.0.smithi163.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_snapshot": "", 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_max": 1, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "max_osd": 1, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "require_min_compat_client": "luminous", 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "min_compat_client": "jewel", 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "require_osd_release": "squid", 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "allow_crimson": false, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "pools": [ 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_name": "rbd", 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "create_time": "2024-05-06T04:22:48.494334+0000", 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "flags": 8193, 2024-05-06T04:23:01.495 INFO:tasks.workunit.client.0.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "type": 1, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "size": 2, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "min_size": 1, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_rule": 0, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "object_hash": 2, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_autoscale_mode": "off", 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_num": 4, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_placement_num": 4, 2024-05-06T04:23:01.496 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_placement_num_target": 4, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_num_target": 4, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_num_pending": 4, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_pg_merge_meta": { 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "source_pgid": "0.0", 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "ready_epoch": 0, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_epoch_started": 0, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_epoch_clean": 0, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "source_version": "0'0", 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "target_version": "0'0" 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_change": "12", 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_force_op_resend": "0", 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-05-06T04:23:01.497 INFO:tasks.workunit.client.0.smithi163.stdout: "auid": 0, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_mode": "selfmanaged", 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_seq": 2, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_epoch": 12, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_snaps": [], 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "removed_snaps": "[]", 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "quota_max_bytes": 0, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "quota_max_objects": 0, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "tiers": [], 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "tier_of": -1, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "read_tier": -1, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "write_tier": -1, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_mode": "none", 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "target_max_bytes": 0, 2024-05-06T04:23:01.498 INFO:tasks.workunit.client.0.smithi163.stdout: "target_max_objects": 0, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_min_flush_age": 0, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "cache_min_evict_age": 0, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "erasure_code_profile": "", 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_params": { 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "none" 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_period": 0, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_count": 0, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "use_gmt_hitset": true, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-05-06T04:23:01.499 INFO:tasks.workunit.client.0.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "hit_set_search_last_n": 0, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "grade_table": [], 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "stripe_width": 0, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "expected_num_objects": 0, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "fast_read": false, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "options": {}, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "application_metadata": { 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "rbd": {} 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "read_balance": { 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "score_type": "Fair distribution", 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "score_acting": 1, 2024-05-06T04:23:01.500 INFO:tasks.workunit.client.0.smithi163.stdout: "score_stable": 1, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "optimal_score": 1, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "raw_score_acting": 1, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "raw_score_stable": 1, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "primary_affinity_weighted": 1, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "average_primary_affinity": 1, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "average_primary_affinity_weighted": 1 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "osds": [ 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "uuid": "e9b56e9e-199b-4395-8b4e-489e042a96dd", 2024-05-06T04:23:01.501 INFO:tasks.workunit.client.0.smithi163.stdout: "up": 1, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "in": 1, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 1, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "primary_affinity": 1, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "last_clean_begin": 0, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "last_clean_end": 0, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "up_from": 5, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "up_thru": 7, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "down_at": 0, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "lost_at": 0, 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addrs": { 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.502 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6800", 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6801", 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_addrs": { 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:23:01.503 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6802", 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6803", 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_back_addrs": { 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6806", 2024-05-06T04:23:01.504 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6807", 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_front_addrs": { 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "addrvec": [ 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v2", 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6804", 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.505 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "v1", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "addr": "127.0.0.1:6805", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "nonce": 275213922 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "public_addr": "127.0.0.1:6801/275213922", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "cluster_addr": "127.0.0.1:6803/275213922", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_back_addr": "127.0.0.1:6807/275213922", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "heartbeat_front_addr": "127.0.0.1:6805/275213922", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "state": [ 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "exists", 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: "up" 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.506 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_xinfo": [ 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "down_stamp": "0.000000", 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "laggy_probability": 0, 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "laggy_interval": 0, 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "features": 4540701547738038271, 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "old_weight": 0, 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "last_purged_snaps_scrub": "2024-05-06T04:22:46.852986+0000", 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "dead_epoch": 0 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_upmap": [], 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_upmap_items": [], 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_upmap_primaries": [], 2024-05-06T04:23:01.507 INFO:tasks.workunit.client.0.smithi163.stdout: "pg_temp": [], 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "primary_temp": [], 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "blocklist": {}, 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "range_blocklist": {}, 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "erasure_code_profiles": { 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "default": { 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "k": "2", 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "m": "2", 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "plugin": "jerasure", 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "technique": "reed_sol_van" 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "removed_snaps_queue": [ 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: "snaps": [ 2024-05-06T04:23:01.508 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "begin": 2, 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "length": 1 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "new_removed_snaps": [ 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "snaps": [ 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "begin": 2, 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: "length": 1 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.509 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "new_purged_snaps": [], 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "crush_node_flags": {}, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "device_class_flags": {}, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode": { 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode_enabled": false, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_bucket_count": 0, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "degraded_stretch_mode": 0, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "recovering_stretch_mode": 0, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "stretch_mode_bucket": 0 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_metadata": [ 2024-05-06T04:23:01.510 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "arch": "x86_64", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "back_addr": "[v2:127.0.0.1:6802/275213922,v1:127.0.0.1:6803/275213922]", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "back_iface": "", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs": "1", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs_dedicated_db": "0", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs_dedicated_wal": "0", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluefs_single_shared_device": "1", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_allocation_from_file": "0", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_access_mode": "file", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_block_size": "4096", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_devices": "sda", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-05-06T04:23:01.511 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_rotational": "1", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_size": "107374182400", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_support_discard": "1", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_bdev_type": "hdd", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "bluestore_min_alloc_size": "4096", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_release": "squid", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_version": "ceph version 19.0.0-3516-ge08cbbf6 (e08cbbf6fd79a8dd2949f340bce25e28ea3db004) squid (dev)", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_version_short": "19.0.0-3516-ge08cbbf6", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "ceph_version_when_created": "ceph version 19.0.0-3516-ge08cbbf6 (e08cbbf6fd79a8dd2949f340bce25e28ea3db004) squid (dev)", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "created_at": "2024-05-06T04:22:22.413344Z", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "default_device_class": "hdd", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFMX", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "devices": "sda", 2024-05-06T04:23:01.512 INFO:tasks.workunit.client.0.smithi163.stdout: "distro": "centos", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "distro_description": "CentOS Stream 9", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "distro_version": "9", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "front_addr": "[v2:127.0.0.1:6800/275213922,v1:127.0.0.1:6801/275213922]", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "front_iface": "", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/275213922,v1:127.0.0.1:6807/275213922]", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/275213922,v1:127.0.0.1:6805/275213922]", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "hostname": "smithi163", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "journal_rotational": "1", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "kernel_version": "5.14.0-437.el9.x86_64", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "mem_swap_kb": "0", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "mem_total_kb": "32488420", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "objectstore_numa_unknown_devices": "sda", 2024-05-06T04:23:01.513 INFO:tasks.workunit.client.0.smithi163.stdout: "os": "Linux", 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_data": "td/ceph-helpers/0", 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_objectstore": "bluestore", 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "osdspec_affinity": "", 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "rotational": "1" 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap_clean_epochs": { 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "min_last_epoch_clean": 0, 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "last_epoch_clean": { 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "per_pool": [] 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_epochs": [ 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:23:01.514 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 5 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap_first_committed": 1, 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "osdmap_last_committed": 12, 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "crushmap": { 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "devices": [ 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "osd.0", 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "class": "hdd" 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.515 INFO:tasks.workunit.client.0.smithi163.stdout: "types": [ 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 0, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "osd" 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 1, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "host" 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 2, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "chassis" 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 3, 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "rack" 2024-05-06T04:23:01.516 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 4, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "row" 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 5, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "pdu" 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 6, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "pod" 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 7, 2024-05-06T04:23:01.517 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "room" 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 8, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "datacenter" 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 9, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "zone" 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 10, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "region" 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.518 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 11, 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "root" 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "buckets": [ 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -1, 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "default", 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 11, 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "root", 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.519 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -3, 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -2, 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "default~hdd", 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 11, 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "root", 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:23:01.520 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -4, 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -3, 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "smithi163", 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 1, 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "host", 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:23:01.521 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "id": -4, 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "name": "smithi163~hdd", 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "type_id": 1, 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "type_name": "host", 2024-05-06T04:23:01.522 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "alg": "straw2", 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "hash": "rjenkins1", 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "items": [ 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "id": 0, 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "weight": 6402, 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "pos": 0 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "rules": [ 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.523 INFO:tasks.workunit.client.0.smithi163.stdout: "rule_id": 0, 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "rule_name": "replicated_rule", 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "type": 1, 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "steps": [ 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "op": "take", 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "item": -1, 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "item_name": "default" 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "op": "choose_firstn", 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "num": 0, 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "type": "osd" 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.524 INFO:tasks.workunit.client.0.smithi163.stdout: "op": "emit" 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: ] 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "tunables": { 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_local_tries": 0, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_local_fallback_tries": 0, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_total_tries": 50, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "chooseleaf_descend_once": 1, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "chooseleaf_vary_r": 1, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "chooseleaf_stable": 1, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "msr_descents": 100, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "msr_collision_tries": 100, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "straw_calc_version": 1, 2024-05-06T04:23:01.525 INFO:tasks.workunit.client.0.smithi163.stdout: "allowed_bucket_algs": 54, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "profile": "jewel", 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "optimal_tunables": 1, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "legacy_tunables": 0, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "minimum_required_version": "jewel", 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables": 1, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables2": 1, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v2_rules": 0, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables3": 1, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v3_rules": 0, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v4_buckets": 1, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "require_feature_tunables5": 1, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "has_v5_rules": 0, 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: "has_msr_rules": 0 2024-05-06T04:23:01.526 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "choose_args": {} 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "fsmap": { 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 1, 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "default_fscid": -1, 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "compat": { 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "compat": {}, 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "ro_compat": {}, 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "incompat": { 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_1": "base v0.20", 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_2": "client writeable ranges", 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_3": "default file layouts on dirs", 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_4": "dir inode in separate object", 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_5": "mds uses versioned encoding", 2024-05-06T04:23:01.527 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_6": "dirfrag is stored in omap", 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_8": "no anchor table", 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_9": "file layout v2", 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_10": "snaprealm v2", 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_11": "minor log segments", 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_12": "quiesce subvolumes" 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "feature_flags": { 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "enable_multiple": true, 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "ever_enabled_multiple": true 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "standbys": [], 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "filesystems": [] 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.528 INFO:tasks.workunit.client.0.smithi163.stdout: "mdsmap_first_committed": 1, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "mdsmap_last_committed": 1, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "auth": { 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "first_committed": 1, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "last_committed": 3, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_secrets": 1 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg": 4, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_active": 4, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_unknown": 0, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_osd": 1, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_sum": { 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "stat_sum": { 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes": 19, 2024-05-06T04:23:01.529 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects": 1, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_clones": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_copies": 2, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing_on_primary": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_degraded": 1, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_misplaced": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_unfound": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_dirty": 1, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_whiteouts": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read_kb": 0, 2024-05-06T04:23:01.530 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write": 2, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write_kb": 2, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_scrub_errors": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shallow_scrub_errors": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_deep_scrub_errors": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_recovered": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_recovered": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_keys_recovered": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_omap": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_hit_set_archive": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_hit_set_archive": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_kb": 0, 2024-05-06T04:23:01.531 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_kb": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_promote": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_high": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_low": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_some": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_full": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_pinned": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_legacy_snapsets": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_large_omap_objects": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_manifest": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_bytes": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_keys": 0, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_repaired": 0 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "store_stats": { 2024-05-06T04:23:01.532 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 0, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 0 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "log_size": 2, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "ondisk_log_size": 2, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "up": 4, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "acting": 4, 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: "num_store_stats": 0 2024-05-06T04:23:01.533 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_sum": { 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "up_from": 0, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "seq": 0, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pgs": 4, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_osds": 1, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_osds": 1, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_omap_osds": 1, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "kb": 104857600, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used": 26936, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_data": 108, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_omap": 1, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_meta": 26814, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_avail": 104830664, 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "statfs": { 2024-05-06T04:23:01.534 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 107374182400, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 107346599936, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 110592, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 22938, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 1599, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 27457985 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_peers": [], 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_trim_queue_len": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "num_snap_trimming": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shards_repaired": 0, 2024-05-06T04:23:01.535 INFO:tasks.workunit.client.0.smithi163.stdout: "op_queue_age_hist": { 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "histogram": [], 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "upper_bound": 1 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "perf_stat": { 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ms": 0, 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ms": 0, 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ns": 0, 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ns": 0 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "alerts": [], 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "network_ping_times": [] 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_sum_by_class": { 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "hdd": { 2024-05-06T04:23:01.536 INFO:tasks.workunit.client.0.smithi163.stdout: "up_from": 0, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "seq": 0, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pgs": 4, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num_osds": 1, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_osds": 1, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "num_per_pool_omap_osds": 1, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "kb": 104857600, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used": 26936, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_data": 108, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_omap": 1, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_used_meta": 26814, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "kb_avail": 104830664, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "statfs": { 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 107374182400, 2024-05-06T04:23:01.537 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 107346599936, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 110592, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 22938, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 1599, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 27457985 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "hb_peers": [], 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "snap_trim_queue_len": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "num_snap_trimming": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shards_repaired": 0, 2024-05-06T04:23:01.538 INFO:tasks.workunit.client.0.smithi163.stdout: "op_queue_age_hist": { 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "histogram": [], 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "upper_bound": 1 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "perf_stat": { 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ms": 0, 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ms": 0, 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "commit_latency_ns": 0, 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "apply_latency_ns": 0 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "alerts": [], 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "network_ping_times": [] 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.539 INFO:tasks.workunit.client.0.smithi163.stdout: "pool_stats": [ 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "poolid": 1, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg": 4, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "stat_sum": { 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes": 19, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects": 1, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_clones": 0, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_object_copies": 2, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing_on_primary": 0, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_missing": 0, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_degraded": 1, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_misplaced": 0, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_unfound": 0, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_dirty": 1, 2024-05-06T04:23:01.540 INFO:tasks.workunit.client.0.smithi163.stdout: "num_whiteouts": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_read_kb": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write": 2, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_write_kb": 2, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_scrub_errors": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_shallow_scrub_errors": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_deep_scrub_errors": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_recovered": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_recovered": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_keys_recovered": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_omap": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_hit_set_archive": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_bytes_hit_set_archive": 0, 2024-05-06T04:23:01.541 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_kb": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_kb": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_promote": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_high": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_flush_mode_low": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_some": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_evict_mode_full": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_pinned": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_legacy_snapsets": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_large_omap_objects": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_manifest": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_bytes": 0, 2024-05-06T04:23:01.542 INFO:tasks.workunit.client.0.smithi163.stdout: "num_omap_keys": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "num_objects_repaired": 0 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "store_stats": { 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "total": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "available": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "internally_reserved": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "allocated": 4096, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "data_stored": 19, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_allocated": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "data_compressed_original": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "omap_allocated": 0, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "internal_metadata": 0 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.543 INFO:tasks.workunit.client.0.smithi163.stdout: "log_size": 2, 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "ondisk_log_size": 2, 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "up": 4, 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "acting": 4, 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "num_store_stats": 1 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "osd_stats": [ 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "seq": 21474836482 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_by_state": [ 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "state": "active+undersized+degraded", 2024-05-06T04:23:01.544 INFO:tasks.workunit.client.0.smithi163.stdout: "num": 1 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "state": "active+undersized", 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "num": 3 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "num_pg_by_osd": [ 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "osd": 0, 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "num_primary_pg": 4, 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "num_acting_pg": 4, 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "num_up_not_acting_pg": 0 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.545 INFO:tasks.workunit.client.0.smithi163.stdout: "purged_snaps": [ 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: { 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "pool": 1, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "purged_snaps": {} 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: ], 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "servicemap": { 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "epoch": 1, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "modified": "2024-05-06T04:22:16.944793+0000", 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "services": {} 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: }, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "mgrstat_first_committed": 1, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "mgrstat_last_committed": 6, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "logm_first_committed": 1, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "logm_last_committed": 79, 2024-05-06T04:23:01.546 INFO:tasks.workunit.client.0.smithi163.stdout: "paxos": { 2024-05-06T04:23:01.547 INFO:tasks.workunit.client.0.smithi163.stdout: "first_committed": 1, 2024-05-06T04:23:01.547 INFO:tasks.workunit.client.0.smithi163.stdout: "last_committed": 104, 2024-05-06T04:23:01.547 INFO:tasks.workunit.client.0.smithi163.stdout: "last_pn": 0, 2024-05-06T04:23:01.547 INFO:tasks.workunit.client.0.smithi163.stdout: "accepted_pn": 0 2024-05-06T04:23:01.547 INFO:tasks.workunit.client.0.smithi163.stdout: } 2024-05-06T04:23:01.547 INFO:tasks.workunit.client.0.smithi163.stdout:} 2024-05-06T04:23:01.550 INFO:tasks.workunit.client.0.smithi163.stderr:report 3485988216 2024-05-06T04:23:01.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:23:01.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:23:01.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:23:01.553 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 d7ff283a-0cc0-450b-b6c5-2ff6e69ab3aa 2024-05-06T04:23:01.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7ff283a-0cc0-450b-b6c5-2ff6e69ab3aa 2024-05-06T04:23:01.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d7ff283a-0cc0-450b-b6c5-2ff6e69ab3aa' 2024-05-06T04:23:01.554 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:23:01.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlWzhmN/68HhAA/xtiqp9Dte+aLCQcidL8xg== 2024-05-06T04:23:01.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlWzhmN/68HhAA/xtiqp9Dte+aLCQcidL8xg=="}' 2024-05-06T04:23:01.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7ff283a-0cc0-450b-b6c5-2ff6e69ab3aa -i td/ceph-helpers/1/new.json 2024-05-06T04:23:01.988 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:23:02.000 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-05-06T04:23:02.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlWzhmN/68HhAA/xtiqp9Dte+aLCQcidL8xg== --osd-uuid d7ff283a-0cc0-450b-b6c5-2ff6e69ab3aa 2024-05-06T04:23:02.029 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.026+0000 7fd588999640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:02.043 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.041+0000 7fd588999640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:02.044 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.042+0000 7fd588999640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:02.044 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.042+0000 7fd588999640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:23:02.044 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.042+0000 7fd588999640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:23:02.044 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.042+0000 7fd588999640 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-05-06T04:23:02.074 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:02.072+0000 7fd588999640 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-05-06T04:23:07.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-05-06T04:23:07.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:23:07.881 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:23:07.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:23:07.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:23:08.430 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:23:08.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:23:08.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:23:08.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:23:08.431 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:23:08.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:23:08.457 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:08.454+0000 7f8e69fb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:08.458 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:08.456+0000 7f8e69fb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:08.459 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:08.457+0000 7f8e69fb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:08.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:23:08.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:23:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:23:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:23:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:23:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:08.852 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:23:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:23:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:08.973 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:08.971+0000 7f8e69fb8640 -1 Falling back to public interface 2024-05-06T04:23:09.252 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:10.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:10.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:10.255 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:23:10.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:23:10.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:10.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:10.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:11.398 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:11.396+0000 7f8e69fb8640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:23:11.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:11.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:11.696 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:23:11.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:23:11.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:11.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:12.208 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:13.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:13.209 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:13.210 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:23:13.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:23:13.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:13.210 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:13.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:14.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:14.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:14.632 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:23:14.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:23:14.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:14.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:15.038 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:16.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:16.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:16.040 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:23:16.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:23:16.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:16.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:16.574 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:17.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:17.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:17.576 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:23:17.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:23:17.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:17.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:18.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:19.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:19.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:19.007 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:23:19.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:23:19.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:19.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:19.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:20.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:20.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:20.506 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:23:20.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:23:20.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:20.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:20.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:21.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:21.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:21.927 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:23:21.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:23:21.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:21.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:22.355 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:23.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:23.357 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:23.357 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:23:23.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:23:23.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:23.358 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:24.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:25.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:25.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:25.105 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:23:25.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:23:25.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:25.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:25.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:26.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:26.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:26.954 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:23:26.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:23:26.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:26.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:27.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:28.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:28.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:28.734 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:23:28.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:23:28.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:28.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:29.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:30.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:30.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:30.847 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:23:30.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:23:30.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:30.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:31.629 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:32.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:32.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:32.631 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:23:32.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:23:32.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:32.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:33.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:34.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:34.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:34.716 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:23:34.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:23:34.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:34.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:23:35.263 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1814869892,v1:127.0.0.1:6811/1814869892] [v2:127.0.0.1:6812/1814869892,v1:127.0.0.1:6813/1814869892] exists,up d7ff283a-0cc0-450b-b6c5-2ff6e69ab3aa 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:23:35.265 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:23:35.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:23:35.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-05-06T04:23:35.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-05-06T04:23:35.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-05-06T04:23:35.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:23:35.346 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:23:35.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:23:35.836 INFO:tasks.workunit.client.0.smithi163.stderr:1' 2024-05-06T04:23:35.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:23:35.836 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:23:35.836 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:23:35.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836491 2024-05-06T04:23:35.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836491 2024-05-06T04:23:35.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491' 2024-05-06T04:23:35.951 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:23:35.951 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=68719476745 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 68719476745 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491 1-68719476745' 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836491 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:23:36.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:23:36.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836491 2024-05-06T04:23:36.080 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:23:36.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836491 2024-05-06T04:23:36.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-05-06T04:23:36.081 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836491 2024-05-06T04:23:36.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:23:36.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836491 2024-05-06T04:23:36.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:23:36.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-68719476745 2024-05-06T04:23:36.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:23:36.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:23:36.505 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-68719476745 2024-05-06T04:23:36.505 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:23:36.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=68719476745 2024-05-06T04:23:36.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 68719476745' 2024-05-06T04:23:36.507 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 68719476745 2024-05-06T04:23:36.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:23:36.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 68719476745 -lt 68719476745 2024-05-06T04:23:36.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-05-06T04:23:36.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:23:36.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:23:37.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-05-06T04:23:37.590 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-05-06T04:23:38.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-05-06T04:23:38.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-05-06T04:23:38.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:23:38.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:23:38.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-05-06T04:23:38.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-05-06T04:23:38.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-05-06T04:23:38.611 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:23:38.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:23:38.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:23:38.718 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:23:38.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:23:38.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:23:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:23:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:23:38.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:23:38.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:23:38.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:23:38.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:23:38.721 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:23:38.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:23:38.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:23:38.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:23:38.732 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:23:38.732 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:38.732 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:38.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:23:38.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:23:38.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:23:38.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:23:38.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_for_scrub td/ceph-helpers 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_wait_for_scrub: local dir=td/ceph-helpers 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: test_wait_for_scrub: setup td/ceph-helpers 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:23:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:23:38.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:23:38.736 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:23:38.736 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:23:38.736 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:23:38.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:23:38.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:23:38.737 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:23:38.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:23:38.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:23:38.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:23:38.739 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:23:38.739 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:23:38.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:23:38.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:23:38.741 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:23:38.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:38.742 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:38.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:23:38.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:23:38.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:23:38.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:23:38.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:23:38.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:38.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:38.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2042: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:23:38.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:23:38.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:23:38.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:23:38.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:23:39.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:23:39.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:23:39.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:23:39.180 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:23:39.180 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:39.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:39.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:23:39.181 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:23:39.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:23:39.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:23:39.366 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:23:39.366 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:39.366 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:39.366 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:23:39.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:23:39.367 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:23:39.447 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:39.448 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:39.448 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:23:39.448 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:23:39.449 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:23:39.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2043: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-05-06T04:23:39.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:23:39.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:23:39.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:23:39.525 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:23:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:23:39.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:23:40.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:23:40.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:23:40.370 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:23:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2044: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-05-06T04:23:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:23:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:23:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:23:40.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:23:40.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:23:40.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0eb73535-cc34-46a4-ac78-6597a9213579 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0eb73535-cc34-46a4-ac78-6597a9213579' 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 0eb73535-cc34-46a4-ac78-6597a9213579 2024-05-06T04:23:40.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:23:40.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMWzhmAmPhGBAAC6kHqVPl+hHnR2HivukoNA== 2024-05-06T04:23:40.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMWzhmAmPhGBAAC6kHqVPl+hHnR2HivukoNA=="}' 2024-05-06T04:23:40.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0eb73535-cc34-46a4-ac78-6597a9213579 -i td/ceph-helpers/0/new.json 2024-05-06T04:23:40.690 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:23:40.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:23:40.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMWzhmAmPhGBAAC6kHqVPl+hHnR2HivukoNA== --osd-uuid 0eb73535-cc34-46a4-ac78-6597a9213579 2024-05-06T04:23:40.730 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.728+0000 7f6a43a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:40.757 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.755+0000 7f6a43a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:40.759 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.756+0000 7f6a43a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:40.759 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.757+0000 7f6a43a12640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:23:40.759 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.757+0000 7f6a43a12640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:23:40.759 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.757+0000 7f6a43a12640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:23:40.797 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:40.795+0000 7f6a43a12640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:23:46.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:23:46.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:23:46.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:23:46.347 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:23:46.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:23:46.899 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:23:46.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:23:46.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:23:46.900 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:23:46.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:23:46.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:23:46.926 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:46.923+0000 7f99ca815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:46.927 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:46.925+0000 7f99ca815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:46.928 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:46.926+0000 7f99ca815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:47.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:23:47.351 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:23:47.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:47.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:47.465 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:47.463+0000 7f99ca815640 -1 Falling back to public interface 2024-05-06T04:23:47.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:48.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:48.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:48.785 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:23:48.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:23:48.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:48.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:49.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:49.905 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:23:49.903+0000 7f99ca815640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:23:50.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:50.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:50.228 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:23:50.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:23:50.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:50.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:50.679 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:51.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:51.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:51.681 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:23:51.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:23:51.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:51.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:52.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:53.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:53.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:53.103 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:23:53.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:23:53.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:53.104 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:53.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:54.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:54.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:54.508 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:23:54.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:23:54.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:54.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:55.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:56.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:56.048 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:56.048 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:23:56.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:23:56.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:56.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:56.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:57.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:57.473 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:57.473 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:23:57.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:23:57.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:57.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:58.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:23:59.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:23:59.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:23:59.042 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:23:59.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:23:59.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:23:59.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:23:59.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:00.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:00.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:00.457 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:24:00.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:24:00.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:00.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:00.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:01.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:01.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:01.895 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:24:01.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:24:01.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:01.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:02.300 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:03.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:03.301 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:03.301 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:24:03.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:24:03.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:03.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:03.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:04.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:04.905 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:24:04.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:04.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:24:04.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:04.906 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:06.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:07.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:07.014 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:07.014 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:24:07.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:24:07.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:07.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:08.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:09.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:09.173 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:09.174 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:24:09.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:24:09.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:09.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:09.790 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:10.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:10.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:10.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:24:10.792 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:24:10.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:10.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:11.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:12.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:12.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:12.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:24:12.528 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:24:12.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:12.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:13.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:14.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:14.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:14.189 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:24:14.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:24:14.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:14.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:14.592 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1759127128,v1:127.0.0.1:6803/1759127128] [v2:127.0.0.1:6804/1759127128,v1:127.0.0.1:6805/1759127128] exists,up 0eb73535-cc34-46a4-ac78-6597a9213579 2024-05-06T04:24:14.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:24:14.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:24:14.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:24:14.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: test_wait_for_scrub: create_rbd_pool 2024-05-06T04:24:14.593 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:24:15.008 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:24:15.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:24:15.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:24:15.512 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:24:15.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:24:16.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:24:17.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_wait_for_scrub: wait_for_clean 2024-05-06T04:24:17.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:24:17.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:24:17.579 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:24:17.579 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:24:17.580 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:24:17.580 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:24:17.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:24:17.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:24:17.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:24:17.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:24:17.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:24:17.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:24:17.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:24:17.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:24:17.672 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:24:18.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:24:18.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:24:18.069 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:24:18.070 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:24:18.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836490 2024-05-06T04:24:18.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836490 2024-05-06T04:24:18.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836490' 2024-05-06T04:24:18.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:24:18.185 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836490 2024-05-06T04:24:18.185 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:24:18.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:24:18.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836490 2024-05-06T04:24:18.186 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:24:18.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836490 2024-05-06T04:24:18.187 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-05-06T04:24:18.188 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836490 2024-05-06T04:24:18.188 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:24:18.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836490 2024-05-06T04:24:18.589 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:24:18.589 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:24:18.589 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:24:19.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:24:19.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:24:19.124 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:24:19.124 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:24:19.124 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:24:19.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:24:19.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:24:19.125 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:24:19.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:24:19.555 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:24:19.555 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:24:19.555 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:24:20.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:24:20.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:24:20.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:24:20.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_wait_for_scrub: local pgid=1.0 2024-05-06T04:24:20.105 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: test_wait_for_scrub: ceph pg repair 1.0 2024-05-06T04:24:20.485 INFO:tasks.workunit.client.0.smithi163.stderr:instructing pg 1.0 on osd.0 to repair 2024-05-06T04:24:20.498 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-05-06T04:24:20.498 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:20.498 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:20.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:20.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:20.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_wait_for_scrub: local last_scrub=2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:20.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: test_wait_for_scrub: wait_for_scrub 1.0 2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:20.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:24:20.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:20.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:24:20.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:24:20.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:24:20.913 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:24:20.913 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:20.913 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:20.913 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:20.913 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:21.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:24:15.345110+0000 '>' 2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:21.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:24:22.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:24:22.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:24:22.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:24:22.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:22.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:22.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:22.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:22.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:24:15.345110+0000 '>' 2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:22.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:24:23.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:24:23.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:24:23.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:24:23.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:23.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:23.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:23.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:24.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:24:15.345110+0000 '>' 2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:24.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:24:25.093 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:24:25.094 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 300 )) 2024-05-06T04:24:25.094 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:24:25.094 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:25.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:25.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:25.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:25.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:24:20.804603+0000 '>' 2024-05-06T04:24:15.345110+0000 2024-05-06T04:24:25.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: wait_for_scrub: return 0 2024-05-06T04:24:25.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-05-06T04:24:25.500 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:24:25.500 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:24:25.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:24:25.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:24:25.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:24:25.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:24:25.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-05-06T04:24:25.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:25.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:25.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:25.605 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:25.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: test_wait_for_scrub: last_scrub=2024-05-06T04:24:20.804603+0000 2024-05-06T04:24:25.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: test_wait_for_scrub: TIMEOUT=1 2024-05-06T04:24:25.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: test_wait_for_scrub: wait_for_scrub 1.0 2024-05-06T04:24:20.804603+0000 2024-05-06T04:24:25.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: local pgid=1.0 2024-05-06T04:24:25.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: wait_for_scrub: local last_scrub=2024-05-06T04:24:20.804603+0000 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: wait_for_scrub: local sname=last_scrub_stamp 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i=0 )) 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-05-06T04:24:25.994 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-05-06T04:24:26.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: wait_for_scrub: test 2024-05-06T04:24:20.804603+0000 '>' 2024-05-06T04:24:20.804603+0000 2024-05-06T04:24:26.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: wait_for_scrub: sleep 1 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i++ )) 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: wait_for_scrub: (( i < 1 )) 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: wait_for_scrub: return 1 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: test_wait_for_scrub: teardown td/ceph-helpers 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:24:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:24:27.378 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:24:27.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:24:27.484 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:24:27.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:24:27.485 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:24:27.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:24:27.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:24:27.486 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:24:27.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:24:27.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:24:27.487 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:24:27.487 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:24:27.488 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:24:27.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:24:27.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:24:27.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:24:27.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:27.496 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:27.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:24:27.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:24:27.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:24:27.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: run_tests: for func in $funcs 2024-05-06T04:24:27.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:24:27.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:24:27.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:24:27.500 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:24:27.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:24:27.501 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:24:27.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:24:27.502 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:24:27.502 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:24:27.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:24:27.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:24:27.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:24:27.504 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:24:27.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:24:27.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:24:27.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:24:27.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:24:27.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:27.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:27.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:24:27.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:24:27.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:24:27.508 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-05-06T04:24:27.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:24:27.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:27.509 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:27.509 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.75593 2024-05-06T04:24:27.511 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:24:27.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-05-06T04:24:27.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:27.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:24:27.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-05-06T04:24:28.172 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:24:28.173 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:24:28.174 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:24:28.174 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:24:28.175 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:28.175 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:28.175 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:24:28.175 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:24:28.175 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get fsid 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:24:28.262 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:24:28.263 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:28.263 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:28.263 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.75593/ceph-mon.a.asok 2024-05-06T04:24:28.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:24:28.263 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.75593/ceph-mon.a.asok config get mon_host 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-05-06T04:24:28.342 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:24:29.194 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:24:29.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-05-06T04:24:29.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:24:29.227 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:24:29.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:24:29.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:24:29.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-05-06T04:24:29.229 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:24:29.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7122148-6594-4eed-8246-6a5ac8f97aa9 2024-05-06T04:24:29.229 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 d7122148-6594-4eed-8246-6a5ac8f97aa9 2024-05-06T04:24:29.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d7122148-6594-4eed-8246-6a5ac8f97aa9' 2024-05-06T04:24:29.230 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:24:29.245 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9WzhmXIVzDhAAqpN8UJzV/HdCxChRPOcMeA== 2024-05-06T04:24:29.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9WzhmXIVzDhAAqpN8UJzV/HdCxChRPOcMeA=="}' 2024-05-06T04:24:29.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7122148-6594-4eed-8246-6a5ac8f97aa9 -i td/ceph-helpers/0/new.json 2024-05-06T04:24:29.513 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:24:29.523 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-05-06T04:24:29.524 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9WzhmXIVzDhAAqpN8UJzV/HdCxChRPOcMeA== --osd-uuid d7122148-6594-4eed-8246-6a5ac8f97aa9 2024-05-06T04:24:29.553 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.551+0000 7f0457e98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:24:29.573 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.571+0000 7f0457e98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:24:29.574 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.572+0000 7f0457e98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:24:29.574 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.572+0000 7f0457e98640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:24:29.574 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.572+0000 7f0457e98640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:24:29.575 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.572+0000 7f0457e98640 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-05-06T04:24:29.613 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:29.611+0000 7f0457e98640 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-05-06T04:24:35.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-05-06T04:24:35.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:24:35.346 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:24:35.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:24:35.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:24:35.906 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:24:35.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:24:35.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=fc6d983f-746c-4b93-98c2-58f795783e09 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.75593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:24:35.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:24:35.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:24:35.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:24:35.932 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:35.930+0000 7efe4b625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:24:35.939 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:35.937+0000 7efe4b625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:24:35.940 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:35.938+0000 7efe4b625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:24:36.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:24:36.324 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:24:36.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:36.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:36.469 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:36.467+0000 7efe4b625640 -1 Falling back to public interface 2024-05-06T04:24:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:37.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:37.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:37.756 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:24:37.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:24:37.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:37.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:38.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:38.809 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:24:38.807+0000 7efe4b625640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:24:39.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:39.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:39.190 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:24:39.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:24:39.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:39.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:39.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:40.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:40.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:40.864 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:24:40.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:24:40.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:40.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:41.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:42.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:42.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:42.303 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:24:42.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:24:42.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:42.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:42.716 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:43.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:43.717 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:43.717 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:24:43.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:24:43.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:43.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:44.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:45.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:45.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:45.152 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:24:45.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:24:45.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:45.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:45.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:46.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:46.590 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:46.591 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:24:46.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:24:46.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:46.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:47.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:48.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:48.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:48.012 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:24:48.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:24:48.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:48.012 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:48.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:49.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:49.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:49.520 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:24:49.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:24:49.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:49.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:49.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:50.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:50.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:50.929 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:24:50.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:24:50.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:50.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:51.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:52.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:52.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:52.617 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:24:52.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:24:52.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:52.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:53.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:54.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:54.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:54.032 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:24:54.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:24:54.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:54.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:54.843 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:55.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:55.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:55.847 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:24:55.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:24:55.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:55.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:56.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:57.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:57.394 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:57.394 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:24:57.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:24:57.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:57.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:24:58.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:24:59.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:24:59.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:24:59.620 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:24:59.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:24:59.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:24:59.621 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:00.325 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:01.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:01.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:01.327 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:25:01.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:25:01.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:01.328 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:01.773 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2368586595,v1:127.0.0.1:6803/2368586595] [v2:127.0.0.1:6804/2368586595,v1:127.0.0.1:6805/2368586595] exists,up d7122148-6594-4eed-8246-6a5ac8f97aa9 2024-05-06T04:25:01.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:25:01.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:25:01.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:25:01.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-05-06T04:25:01.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-06T04:25:02.264 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' does not exist 2024-05-06T04:25:02.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-06T04:25:02.275 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-06T04:25:02.772 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'rbd' already exists 2024-05-06T04:25:02.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:25:03.785 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-06T04:25:04.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-05-06T04:25:04.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:25:04.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:25:04.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:25:04.837 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:25:04.838 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:25:04.838 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:25:04.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:25:04.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:25:04.838 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:25:04.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:25:04.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:25:04.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:25:04.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:25:04.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:25:04.930 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:25:05.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-05-06T04:25:05.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:25:05.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:25:05.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:25:05.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-05-06T04:25:05.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-05-06T04:25:05.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-05-06T04:25:05.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:25:05.480 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-05-06T04:25:05.480 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:25:05.481 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:25:05.481 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-05-06T04:25:05.481 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:25:05.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-05-06T04:25:05.482 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836487 2024-05-06T04:25:05.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-05-06T04:25:05.483 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:25:05.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-05-06T04:25:05.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:25:06.884 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:25:06.884 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:25:07.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836487 2024-05-06T04:25:07.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:25:08.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-05-06T04:25:08.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:25:08.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836487 2024-05-06T04:25:08.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:25:08.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:25:08.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:25:09.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:25:09.369 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:25:09.767 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-06T04:25:09.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:25:09.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:25:09.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-05-06T04:25:10.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-05-06T04:25:10.304 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:25:10.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-05-06T04:25:10.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-05-06T04:25:10.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-05-06T04:25:10.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-05-06T04:25:10.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-05-06T04:25:10.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-05-06T04:25:10.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-05-06T04:25:10.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-05-06T04:25:10.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-05-06T04:25:10.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:25:10.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:25:10.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:25:10.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:25:10.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:25:10.824 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:25:10.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:25:10.825 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:25:10.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:25:10.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:25:10.827 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:25:10.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:25:10.827 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:25:10.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:25:10.828 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:25:10.828 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:25:10.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:25:10.829 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-05-06T04:25:10.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:25:10.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:10.835 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.75593 2024-05-06T04:25:10.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.75593 2024-05-06T04:25:10.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:25:10.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:25:10.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: : exit 0 2024-05-06T04:25:10.837 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-06T04:25:10.838 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T04:25:10.901 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-05-06T04:25:10.901 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-05-06T04:25:10.961 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-05-06T04:25:10.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/test-snaptrim-stats 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none ' 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-05-06T04:25:10.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:25:10.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:25:10.972 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:25:10.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:25:10.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:25:10.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:25:10.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:25:10.976 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:25:10.976 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:10.976 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:10.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:25:10.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:25:10.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:25:10.977 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-05-06T04:25:10.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:25:10.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:10.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:10.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.172593 2024-05-06T04:25:10.982 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-05-06T04:25:10.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:25:10.984 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:25:10.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:25:10.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:25:10.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:25:10.987 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:25:10.987 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:10.987 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:10.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:25:10.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:25:10.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:25:10.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-05-06T04:25:10.989 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:25:10.989 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:10.989 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:10.989 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.172593 2024-05-06T04:25:10.991 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:25:10.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:25:10.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-05-06T04:25:10.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-05-06T04:25:11.470 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:25:11.470 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:25:11.470 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:25:11.470 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:25:11.470 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:11.471 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:11.471 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:25:11.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-05-06T04:25:11.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:25:11.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:25:11.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:25:11.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:25:11.665 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:25:11.666 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:25:11.666 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:25:11.666 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:25:11.666 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:25:11.666 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:25:11.667 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:11.667 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:11.667 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.172593/ceph-mon.a.asok 2024-05-06T04:25:11.667 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:25:11.668 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.172593/ceph-mon.a.asok config get fsid 2024-05-06T04:25:11.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:25:11.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:25:11.744 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.172593/ceph-mon.a.asok 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:25:11.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.172593/ceph-mon.a.asok config get mon_host 2024-05-06T04:25:11.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-05-06T04:25:11.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-05-06T04:25:11.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:25:11.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:25:11.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:25:11.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-05-06T04:25:11.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:25:12.687 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:25:12.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:25:12.712 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-05-06T04:25:12.713 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-05-06T04:25:12.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-05-06T04:25:12.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-05-06T04:25:12.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:25:12.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:25:12.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:25:12.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-05-06T04:25:12.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-05-06T04:25:12.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:25:12.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:25:12.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-05-06T04:25:12.721 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:25:12.721 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:25:12.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ccd5b641-5e74-4070-8bc7-8784a76cfc23 2024-05-06T04:25:12.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ccd5b641-5e74-4070-8bc7-8784a76cfc23' 2024-05-06T04:25:12.724 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 ccd5b641-5e74-4070-8bc7-8784a76cfc23 2024-05-06T04:25:12.724 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:25:12.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoWzhmFPyjKxAAcMX3iXlmMQ6xN37XdOmDxA== 2024-05-06T04:25:12.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoWzhmFPyjKxAAcMX3iXlmMQ6xN37XdOmDxA=="}' 2024-05-06T04:25:12.734 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ccd5b641-5e74-4070-8bc7-8784a76cfc23 -i td/test-snaptrim-stats/0/new.json 2024-05-06T04:25:12.996 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:25:13.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-05-06T04:25:13.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCoWzhmFPyjKxAAcMX3iXlmMQ6xN37XdOmDxA== --osd-uuid ccd5b641-5e74-4070-8bc7-8784a76cfc23 2024-05-06T04:25:13.034 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.032+0000 7fc7075b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:13.064 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.062+0000 7fc7075b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.063+0000 7fc7075b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:13.065 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.064+0000 7fc7075b2640 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-05-06T04:25:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.064+0000 7fc7075b2640 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-05-06T04:25:13.066 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.064+0000 7fc7075b2640 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-05-06T04:25:13.104 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:13.102+0000 7fc7075b2640 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-05-06T04:25:18.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-05-06T04:25:18.664 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:25:18.664 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:25:18.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:25:18.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:25:19.260 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:25:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:25:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:25:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:25:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:25:19.261 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:25:19.285 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:19.283+0000 7f0ed4021640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:19.287 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:19.285+0000 7f0ed4021640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:19.288 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:19.286+0000 7f0ed4021640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:19.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:25:19.653 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:25:19.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:19.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:19.803 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:19.801+0000 7f0ed4021640 -1 Falling back to public interface 2024-05-06T04:25:20.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:21.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:21.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:21.074 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:25:21.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:25:21.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:21.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:21.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:22.201 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:22.199+0000 7f0ed4021640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:25:22.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:22.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:22.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:25:22.507 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:25:22.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:22.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:22.959 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:23.960 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:23.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:23.961 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:25:23.961 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:25:23.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:23.962 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:24.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:25.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:25.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:25.367 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:25:25.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:25:25.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:25.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:25.769 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:26.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:26.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:26.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:25:26.771 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:25:26.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:26.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:27.281 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:28.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:28.282 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:28.282 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:25:28.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:25:28.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:28.283 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:28.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:29.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:29.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:29.687 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:25:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:25:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:29.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:30.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:31.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:31.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:31.121 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:25:31.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:25:31.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:31.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:31.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:32.544 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:32.544 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:25:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:25:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:32.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:33.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:33.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:33.987 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:25:33.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:25:33.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:33.988 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:34.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:35.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:35.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:35.550 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:25:35.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:25:35.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:35.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:36.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:37.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:37.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:37.227 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:25:37.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:25:37.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:37.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:38.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:39.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:39.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:39.076 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:25:39.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:25:39.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:39.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:39.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:40.989 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:40.989 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:40.989 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:25:40.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:25:40.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:40.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:41.459 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:42.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:42.460 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:25:42.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:42.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:25:42.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:42.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:43.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:44.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:44.367 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:44.367 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:25:44.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:25:44.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:44.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:45.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:46.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:46.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:46.318 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:25:46.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:25:46.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:46.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:25:46.724 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/552901717,v1:127.0.0.1:6803/552901717] [v2:127.0.0.1:6804/552901717,v1:127.0.0.1:6805/552901717] exists,up ccd5b641-5e74-4070-8bc7-8784a76cfc23 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:25:46.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:25:46.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-05-06T04:25:46.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:25:46.731 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 13c7d3f7-2f87-4ea6-872d-f645642da284 2024-05-06T04:25:46.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13c7d3f7-2f87-4ea6-872d-f645642da284 2024-05-06T04:25:46.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 13c7d3f7-2f87-4ea6-872d-f645642da284' 2024-05-06T04:25:46.732 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:25:46.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKWzhmjY05LBAAN+PkStkskKmwBtNZg7Rxhw== 2024-05-06T04:25:46.743 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKWzhmjY05LBAAN+PkStkskKmwBtNZg7Rxhw=="}' 2024-05-06T04:25:46.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13c7d3f7-2f87-4ea6-872d-f645642da284 -i td/test-snaptrim-stats/1/new.json 2024-05-06T04:25:47.216 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:25:47.227 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-05-06T04:25:47.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDKWzhmjY05LBAAN+PkStkskKmwBtNZg7Rxhw== --osd-uuid 13c7d3f7-2f87-4ea6-872d-f645642da284 2024-05-06T04:25:47.255 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.253+0000 7f0831417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:47.266 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.264+0000 7f0831417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:47.267 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.265+0000 7f0831417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:47.267 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.266+0000 7f0831417640 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-05-06T04:25:47.268 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.266+0000 7f0831417640 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-05-06T04:25:47.268 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.266+0000 7f0831417640 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-05-06T04:25:47.306 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:47.305+0000 7f0831417640 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-05-06T04:25:53.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-05-06T04:25:53.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:25:53.318 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:25:53.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:25:53.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:25:53.860 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:25:53.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:25:53.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:25:53.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:25:53.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:25:53.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:25:53.886 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:53.884+0000 7f710c641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:53.891 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:53.889+0000 7f710c641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:53.892 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:53.890+0000 7f710c641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:25:54.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:25:54.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:25:54.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:25:54.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:25:54.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:25:54.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:54.306 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:25:54.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:25:54.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:54.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:25:54.423 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:54.422+0000 7f710c641640 -1 Falling back to public interface 2024-05-06T04:25:54.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:55.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:55.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:55.732 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:25:55.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:25:55.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:55.733 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:25:56.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:56.944 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:25:56.942+0000 7f710c641640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:25:57.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:57.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:57.217 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:25:57.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:25:57.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:57.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:25:57.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:25:58.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:25:58.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:25:58.748 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:25:58.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:25:58.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:25:58.749 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:25:59.152 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:00.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:00.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:00.154 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:26:00.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:26:00.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:00.155 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:00.563 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:01.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:01.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:01.565 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:26:01.565 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:26:01.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:01.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:02.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:03.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:03.077 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:03.077 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:26:03.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:26:03.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:03.078 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:03.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:04.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:04.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:04.499 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:26:04.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:26:04.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:04.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:05.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:06.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:06.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:06.073 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:26:06.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:26:06.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:06.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:06.511 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:07.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:07.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:07.513 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:26:07.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:26:07.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:07.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:07.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:08.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:08.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:08.931 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:26:08.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:26:08.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:08.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:09.330 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:10.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:10.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:10.332 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:26:10.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:26:10.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:10.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:10.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:11.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:11.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:11.814 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:26:11.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:26:11.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:11.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:12.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:13.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:13.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:13.548 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:26:13.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:26:13.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:13.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:14.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:15.673 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:26:15.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:26:15.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:15.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:16.864 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:17.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:17.865 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:17.866 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:26:17.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:26:17.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:17.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:18.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:19.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:19.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:19.501 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:26:19.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:26:19.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:19.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:20.620 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:21.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:21.622 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:26:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:26:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:21.623 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:26:22.020 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4110736721,v1:127.0.0.1:6811/4110736721] [v2:127.0.0.1:6812/4110736721,v1:127.0.0.1:6813/4110736721] exists,up 13c7d3f7-2f87-4ea6-872d-f645642da284 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-05-06T04:26:22.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:26:22.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-05-06T04:26:22.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:26:22.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8318c53-9225-44d9-8088-bd364a6080d9 2024-05-06T04:26:22.027 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 b8318c53-9225-44d9-8088-bd364a6080d9 2024-05-06T04:26:22.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b8318c53-9225-44d9-8088-bd364a6080d9' 2024-05-06T04:26:22.027 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:26:22.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuWzhmcfVbAhAAWYdtK2MTBI1RAt38wQWiFw== 2024-05-06T04:26:22.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuWzhmcfVbAhAAWYdtK2MTBI1RAt38wQWiFw=="}' 2024-05-06T04:26:22.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8318c53-9225-44d9-8088-bd364a6080d9 -i td/test-snaptrim-stats/2/new.json 2024-05-06T04:26:22.541 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:26:22.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-05-06T04:26:22.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDuWzhmcfVbAhAAWYdtK2MTBI1RAt38wQWiFw== --osd-uuid b8318c53-9225-44d9-8088-bd364a6080d9 2024-05-06T04:26:22.580 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.579+0000 7f8b6b415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:26:22.582 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.580+0000 7f8b6b415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:26:22.583 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.581+0000 7f8b6b415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:26:22.583 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.582+0000 7f8b6b415640 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-05-06T04:26:22.583 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.582+0000 7f8b6b415640 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-05-06T04:26:22.583 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.582+0000 7f8b6b415640 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-05-06T04:26:22.665 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:22.664+0000 7f8b6b415640 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-05-06T04:26:28.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-05-06T04:26:28.665 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:26:28.666 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:26:28.667 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:26:28.667 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:26:29.222 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:26:29.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:26:29.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:26:29.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:26:29.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:26:29.223 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:26:29.248 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:29.246+0000 7f49ccc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:26:29.258 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:29.256+0000 7f49ccc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:26:29.259 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:29.257+0000 7f49ccc12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:29.722 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:26:29.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:26:29.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:29.723 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:29.794 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:29.793+0000 7f49ccc12640 -1 Falling back to public interface 2024-05-06T04:26:30.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:31.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:31.133 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:26:31.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:31.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:26:31.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:31.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:31.548 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:32.286 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:26:32.284+0000 7f49ccc12640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:26:32.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:32.550 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:26:32.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:26:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:33.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:34.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:34.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:34.216 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:26:34.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:26:34.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:34.217 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:34.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:35.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:35.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:35.731 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:26:35.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:26:35.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:35.732 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:36.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:37.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:37.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:37.230 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:26:37.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:26:37.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:37.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:37.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:38.631 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:38.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:38.632 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:26:38.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:26:38.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:38.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:39.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:40.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:40.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:40.145 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:26:40.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:26:40.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:40.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:40.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:41.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:41.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:41.605 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:26:41.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:26:41.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:41.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:42.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:43.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:43.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:43.044 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:26:43.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:26:43.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:43.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:43.481 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:44.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:44.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:44.483 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:26:44.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:26:44.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:44.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:45.072 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:46.073 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:46.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:46.074 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:26:46.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:26:46.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:46.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:46.942 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:47.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:47.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:47.944 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:26:47.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:26:47.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:47.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:48.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:49.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:49.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:49.772 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:26:49.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:26:49.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:49.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:50.578 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:51.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:51.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:51.580 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:26:51.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:26:51.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:51.581 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:52.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:53.633 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:53.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:53.634 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:26:53.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:26:53.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:53.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:54.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:26:55.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:26:55.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:26:55.476 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:26:55.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:26:55.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:26:55.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:26:55.879 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2094000621,v1:127.0.0.1:6819/2094000621] [v2:127.0.0.1:6820/2094000621,v1:127.0.0.1:6821/2094000621] exists,up b8318c53-9225-44d9-8088-bd364a6080d9 2024-05-06T04:26:55.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:26:55.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:26:55.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:26:55.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-05-06T04:26:56.811 INFO:tasks.workunit.client.0.smithi163.stderr:noscrub is set 2024-05-06T04:26:56.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-05-06T04:26:57.607 INFO:tasks.workunit.client.0.smithi163.stderr:nodeep-scrub is set 2024-05-06T04:26:57.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-05-06T04:26:57.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-05-06T04:26:58.122 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'test' already exists 2024-05-06T04:26:58.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:26:59.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-05-06T04:26:59.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:26:59.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:26:59.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:26:59.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:26:59.137 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:26:59.137 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:26:59.137 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:26:59.137 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:26:59.137 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:26:59.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:26:59.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:26:59.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:26:59.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:26:59.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:26:59.228 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:26:59.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:26:59.850 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T04:26:59.850 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T04:26:59.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:26:59.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:26:59.850 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:27:00.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-05-06T04:27:00.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-05-06T04:27:00.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-05-06T04:27:00.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:27:00.002 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:27:00.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-05-06T04:27:00.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-05-06T04:27:00.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-05-06T04:27:00.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:27:00.115 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T04:27:00.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509444 2024-05-06T04:27:00.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509444 2024-05-06T04:27:00.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509444' 2024-05-06T04:27:00.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:27:00.248 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-05-06T04:27:00.249 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:27:00.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:27:00.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-05-06T04:27:00.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:27:00.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-05-06T04:27:00.251 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836502 2024-05-06T04:27:00.251 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-05-06T04:27:00.251 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:27:00.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836502 2024-05-06T04:27:00.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:27:01.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:27:01.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:27:02.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-05-06T04:27:02.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:27:02.135 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-05-06T04:27:02.135 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:27:02.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:27:02.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-05-06T04:27:02.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:27:02.137 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672976 2024-05-06T04:27:02.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-05-06T04:27:02.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-05-06T04:27:02.138 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:27:02.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-05-06T04:27:02.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:27:02.528 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509444 2024-05-06T04:27:02.528 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:27:02.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T04:27:02.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509444 2024-05-06T04:27:02.529 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:27:02.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509444 2024-05-06T04:27:02.530 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509444 2024-05-06T04:27:02.530 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-05-06T04:27:02.531 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T04:27:02.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509445 -lt 64424509444 2024-05-06T04:27:02.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:27:02.921 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:27:02.922 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:27:03.452 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-05-06T04:27:03.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:27:03.453 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:27:03.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:27:03.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:27:03.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:27:03.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:27:03.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:27:03.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-05-06T04:27:03.845 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:27:03.845 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:27:03.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:27:04.382 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-05-06T04:27:04.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:27:04.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:27:04.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-05-06T04:27:04.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-05-06T04:27:04.383 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-05-06T04:27:04.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-05-06T04:27:04.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-05-06T04:27:04.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-05-06T04:27:04.812 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records in 2024-05-06T04:27:04.812 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records out 2024-05-06T04:27:04.813 INFO:tasks.workunit.client.0.smithi163.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.6034e-05 s, 73.1 MB/s 2024-05-06T04:27:04.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-05-06T04:27:04.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:04.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-05-06T04:27:04.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:04.974 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-05-06T04:27:05.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:05.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-05-06T04:27:05.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:05.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-05-06T04:27:05.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:05.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-05-06T04:27:05.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:05.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-05-06T04:27:05.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:05.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-05-06T04:27:05.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:05.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-05-06T04:27:06.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:06.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-05-06T04:27:06.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:06.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-05-06T04:27:06.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-05-06T04:27:06.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-05-06T04:27:06.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-05-06T04:27:06.475 INFO:tasks.workunit.client.0.smithi163.stdout:created pool test snap snap1 2024-05-06T04:27:06.477 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-05-06T04:27:06.477 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-05-06T04:27:06.478 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records in 2024-05-06T04:27:06.478 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records out 2024-05-06T04:27:06.478 INFO:tasks.workunit.client.0.smithi163.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.2489e-05 s, 78.0 MB/s 2024-05-06T04:27:06.478 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-05-06T04:27:06.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:06.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-05-06T04:27:06.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:06.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-05-06T04:27:06.929 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:06.930 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-05-06T04:27:07.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:07.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-05-06T04:27:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-05-06T04:27:07.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:07.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-05-06T04:27:07.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:07.566 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-05-06T04:27:07.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:07.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-05-06T04:27:07.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:07.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-05-06T04:27:08.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-05-06T04:27:08.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-05-06T04:27:08.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-05-06T04:27:08.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-05-06T04:27:08.338 INFO:tasks.workunit.client.0.smithi163.stdout:removed pool test snap snap1 2024-05-06T04:27:08.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-05-06T04:27:08.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:27:08.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:27:08.340 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:27:08.340 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:27:08.341 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:27:08.341 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:27:08.341 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:27:08.341 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:27:08.341 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:27:08.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:27:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:27:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:27:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:27:08.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:27:08.422 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:27:08.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:27:08.825 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T04:27:08.825 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T04:27:08.825 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:27:08.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:27:08.826 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:27:09.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-05-06T04:27:09.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-05-06T04:27:09.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-05-06T04:27:09.112 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:27:09.112 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:27:09.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672980 2024-05-06T04:27:09.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672980 2024-05-06T04:27:09.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672980' 2024-05-06T04:27:09.580 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:27:09.580 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T04:27:09.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-05-06T04:27:09.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-05-06T04:27:09.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672980 2-64424509448' 2024-05-06T04:27:09.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:27:09.692 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-05-06T04:27:09.692 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:27:09.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:27:09.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-05-06T04:27:09.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:27:09.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-05-06T04:27:09.695 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836506 2024-05-06T04:27:09.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-05-06T04:27:09.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:27:10.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836506 2024-05-06T04:27:10.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:27:11.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:27:11.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:27:11.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-05-06T04:27:11.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:27:11.554 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672980 2024-05-06T04:27:11.554 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:27:11.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:27:11.555 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672980 2024-05-06T04:27:11.555 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:27:11.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672980 2024-05-06T04:27:11.556 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-05-06T04:27:11.556 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672980 2024-05-06T04:27:11.557 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:27:12.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672980 2024-05-06T04:27:12.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:27:12.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-05-06T04:27:12.006 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:27:12.006 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T04:27:12.007 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-05-06T04:27:12.007 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:27:12.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-05-06T04:27:12.008 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509448 2024-05-06T04:27:12.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-05-06T04:27:12.009 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T04:27:12.432 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509448 2024-05-06T04:27:12.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:27:12.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:27:12.432 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:27:12.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-05-06T04:27:12.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:27:12.981 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:27:12.981 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:27:12.981 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:27:12.981 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:27:12.982 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:27:12.982 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:27:13.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-05-06T04:27:13.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:27:13.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:27:13.394 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:27:13.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-05-06T04:27:13.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:27:13.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:27:13.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-05-06T04:27:23.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-05-06T04:27:23.928 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-05-06T04:27:23.928 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-05-06T04:27:23.930 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-05-06T04:27:23.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:23.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-05-06T04:27:23.932 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-05-06T04:27:23.932 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:24.059 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-05-06T04:27:24.060 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-05-06T04:27:24.061 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:24.061 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-05-06T04:27:24.062 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:24.179 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.84781936000000002 2024-05-06T04:27:24.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.84781936000000002 2024-05-06T04:27:24.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:24.180 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-05-06T04:27:24.180 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-05-06T04:27:24.180 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:24.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-05-06T04:27:24.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-05-06T04:27:24.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:24.299 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-05-06T04:27:24.299 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:24.415 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .84781936000000002 + 0.60455149100000005 2024-05-06T04:27:24.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.45237085100000007 2024-05-06T04:27:24.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:24.416 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-05-06T04:27:24.417 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-05-06T04:27:24.417 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:24.535 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-05-06T04:27:24.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-05-06T04:27:24.536 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:24.537 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-05-06T04:27:24.537 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:24.654 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.45237085100000007 + 0.00039014799999999999 2024-05-06T04:27:24.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.45276099900000006999 2024-05-06T04:27:24.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:24.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-05-06T04:27:24.655 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-05-06T04:27:24.656 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:24.772 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-05-06T04:27:24.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-05-06T04:27:24.774 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:24.774 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-05-06T04:27:24.775 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:24.891 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.45276099900000006999 + 0.495398897 2024-05-06T04:27:24.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.94815989600000006999 2024-05-06T04:27:24.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:24.892 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-05-06T04:27:24.892 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-05-06T04:27:24.892 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:25.009 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-05-06T04:27:25.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-05-06T04:27:25.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:25.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-05-06T04:27:25.011 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:25.128 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.94815989600000006999 + 0.25402010200000003 2024-05-06T04:27:25.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.20217999800000009999 2024-05-06T04:27:25.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:25.129 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-05-06T04:27:25.129 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-05-06T04:27:25.130 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:25.245 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-05-06T04:27:25.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-05-06T04:27:25.247 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:25.247 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-05-06T04:27:25.247 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:25.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.20217999800000009999 + 0.72260290500000002 2024-05-06T04:27:25.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.92478290300000011999 2024-05-06T04:27:25.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:25.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-05-06T04:27:25.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-05-06T04:27:25.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:25.481 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-05-06T04:27:25.481 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-05-06T04:27:25.482 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:25.483 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-05-06T04:27:25.483 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:25.599 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.92478290300000011999 + 0.25419995099999998 2024-05-06T04:27:25.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.17898285400000009999 2024-05-06T04:27:25.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:27:25.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-05-06T04:27:25.600 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-05-06T04:27:25.600 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-05-06T04:27:25.718 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-05-06T04:27:25.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-05-06T04:27:25.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-05-06T04:27:25.720 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-05-06T04:27:25.720 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-05-06T04:27:25.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 3.17898285400000009999 + 1.078909439 2024-05-06T04:27:25.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=4.25789229300000009999 2024-05-06T04:27:25.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-05-06T04:27:25.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '4.25789229300000009999 > 0.0' 2024-05-06T04:27:25.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-05-06T04:27:25.859 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:27:25.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:27:25.966 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:27:25.966 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:27:25.967 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:27:25.967 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:27:25.968 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:27:25.969 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:27:25.969 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:27:25.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:25.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:27:25.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:27:25.970 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:25.970 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:27:25.971 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:27:25.972 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:27:26.000 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:27:26.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.001 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:27:26.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:27:26.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:27:26.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-05-06T04:27:26.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.004 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:27:26.005 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:27:26.007 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:26.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:27:26.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:27:26.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:27:26.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:27:26.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.010 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:27:26.013 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:27:26.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:27:26.014 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:27:26.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:27:26.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:27:26.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:27:26.015 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:27:26.015 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:27:26.016 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:27:26.017 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:26.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:27:26.017 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:27:26.018 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:27:26.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:27:26.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.019 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:27:26.019 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:27:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:27:26.020 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-05-06T04:27:26.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:27:26.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.021 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.172593 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-05-06T04:27:26.025 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:27:26.026 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:27:26.028 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:27:26.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:27:26.028 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:27:26.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:27:26.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:27:26.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:27:26.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:26.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:27:26.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:27:26.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:27:26.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:27:26.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:27:26.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:27:26.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:27:26.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.031 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:27:26.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:27:26.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:27:26.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-05-06T04:27:26.033 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:27:26.033 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.033 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.172593 2024-05-06T04:27:26.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:27:26.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:27:26.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:27:26.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-05-06T04:27:26.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-05-06T04:27:26.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:27:26.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:27:26.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:27:26.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-05-06T04:27:26.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-05-06T04:27:26.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:27:26.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:27:26.506 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:27:26.506 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:27:26.507 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.507 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.507 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:27:26.507 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-05-06T04:27:26.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.172593/ceph-mon.a.asok 2024-05-06T04:27:26.684 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:27:26.685 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.172593/ceph-mon.a.asok config get fsid 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:27:26.767 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:26.768 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:26.768 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.172593/ceph-mon.a.asok 2024-05-06T04:27:26.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:27:26.768 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.172593/ceph-mon.a.asok config get mon_host 2024-05-06T04:27:26.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-05-06T04:27:26.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-05-06T04:27:26.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:27:26.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:27:26.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:27:26.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-05-06T04:27:26.846 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:27.695 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:27:27.696 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:27:27.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:27:27.722 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-05-06T04:27:27.723 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:27:27.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:27:27.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93efe0f4-854d-4401-b1db-7cb7c22fe13b 2024-05-06T04:27:27.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 93efe0f4-854d-4401-b1db-7cb7c22fe13b' 2024-05-06T04:27:27.731 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 93efe0f4-854d-4401-b1db-7cb7c22fe13b 2024-05-06T04:27:27.731 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:27:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvXDhmTsJCLBAAffE4zJgDbVj9LbB6NXQ91g== 2024-05-06T04:27:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvXDhmTsJCLBAAffE4zJgDbVj9LbB6NXQ91g=="}' 2024-05-06T04:27:27.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93efe0f4-854d-4401-b1db-7cb7c22fe13b -i td/test-snaptrim-stats/0/new.json 2024-05-06T04:27:28.023 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:27:28.032 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-05-06T04:27:28.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAvXDhmTsJCLBAAffE4zJgDbVj9LbB6NXQ91g== --osd-uuid 93efe0f4-854d-4401-b1db-7cb7c22fe13b 2024-05-06T04:27:28.061 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.059+0000 7f1a15106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:27:28.091 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.090+0000 7f1a15106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:27:28.092 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.091+0000 7f1a15106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:27:28.093 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.091+0000 7f1a15106640 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-05-06T04:27:28.093 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.091+0000 7f1a15106640 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-05-06T04:27:28.093 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.091+0000 7f1a15106640 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-05-06T04:27:28.132 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:28.130+0000 7f1a15106640 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-05-06T04:27:33.729 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-05-06T04:27:33.730 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:27:33.730 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:27:33.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:27:33.731 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:27:34.284 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:27:34.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:27:34.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:27:34.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:27:34.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:27:34.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:27:34.310 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:34.309+0000 7fdf7dc0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:27:34.313 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:34.311+0000 7fdf7dc0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:27:34.314 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:34.313+0000 7fdf7dc0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:34.711 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:27:34.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:27:34.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:34.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:34.840 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:34.839+0000 7fdf7dc0e640 -1 Falling back to public interface 2024-05-06T04:27:35.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:36.137 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:36.138 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:27:36.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:36.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:27:36.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:36.138 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:36.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:37.199 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:27:37.198+0000 7fdf7dc0e640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:27:37.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:37.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:37.572 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:27:37.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:27:37.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:37.573 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:37.983 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:38.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:38.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:38.985 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:27:38.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:27:38.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:38.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:39.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:40.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:40.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:40.392 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:27:40.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:27:40.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:40.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:40.792 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:41.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:41.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:41.794 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:27:41.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:27:41.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:41.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:42.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:43.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:43.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:43.345 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:27:43.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:27:43.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:43.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:43.775 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:44.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:44.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:44.777 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:27:44.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:27:44.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:44.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:45.197 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:46.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:46.198 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:46.199 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:27:46.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:27:46.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:46.199 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:46.640 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:47.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:47.641 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:47.642 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:27:47.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:27:47.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:47.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:48.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:49.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:49.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:49.100 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:27:49.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:27:49.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:49.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:49.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:50.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:50.702 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:27:50.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:27:50.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:50.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:51.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:52.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:52.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:52.404 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:27:52.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:27:52.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:52.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:53.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:54.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:54.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:54.123 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:27:54.123 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:27:54.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:54.124 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:54.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:55.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:55.980 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:55.980 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:27:55.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:27:55.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:55.981 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:57.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:27:58.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:27:58.135 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:27:58.135 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:27:58.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:27:58.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:27:58.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:27:59.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:00.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:00.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:00.421 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:28:00.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:28:00.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:00.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:28:00.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:01.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:01.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:01.853 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:28:01.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:28:01.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:01.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:28:02.290 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/296406711,v1:127.0.0.1:6803/296406711] [v2:127.0.0.1:6804/296406711,v1:127.0.0.1:6805/296406711] exists,up 93efe0f4-854d-4401-b1db-7cb7c22fe13b 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-05-06T04:28:02.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:28:02.296 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-05-06T04:28:02.297 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:28:02.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f81b04a-e228-4965-afbb-74d9b871fd65 2024-05-06T04:28:02.298 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 2f81b04a-e228-4965-afbb-74d9b871fd65 2024-05-06T04:28:02.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2f81b04a-e228-4965-afbb-74d9b871fd65' 2024-05-06T04:28:02.298 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:28:02.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSXDhmjadsEhAAiDHorV9QFPomLhGLyMhd7g== 2024-05-06T04:28:02.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSXDhmjadsEhAAiDHorV9QFPomLhGLyMhd7g=="}' 2024-05-06T04:28:02.310 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f81b04a-e228-4965-afbb-74d9b871fd65 -i td/test-snaptrim-stats/1/new.json 2024-05-06T04:28:02.767 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:28:02.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-05-06T04:28:02.779 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBSXDhmjadsEhAAiDHorV9QFPomLhGLyMhd7g== --osd-uuid 2f81b04a-e228-4965-afbb-74d9b871fd65 2024-05-06T04:28:02.807 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.806+0000 7f9e61bfc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:02.809 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.808+0000 7f9e61bfc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:02.810 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.809+0000 7f9e61bfc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:02.811 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.809+0000 7f9e61bfc640 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-05-06T04:28:02.811 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.809+0000 7f9e61bfc640 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-05-06T04:28:02.811 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.809+0000 7f9e61bfc640 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-05-06T04:28:02.848 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:02.847+0000 7f9e61bfc640 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-05-06T04:28:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-05-06T04:28:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:28:08.853 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:28:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:28:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:28:09.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:28:09.403 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:28:09.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:28:09.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:28:09.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:28:09.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:28:09.430 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:09.428+0000 7f1bd781b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:09.449 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:09.447+0000 7f1bd781b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:09.450 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:09.448+0000 7f1bd781b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:09.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:28:09.894 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:28:09.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:28:09.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:28:09.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:28:09.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:09.895 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:28:09.895 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:28:09.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:09.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:09.991 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:09.990+0000 7f1bd781b640 -1 Falling back to public interface 2024-05-06T04:28:10.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:11.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:11.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:11.365 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:28:11.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:28:11.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:11.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:11.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:12.452 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:12.450+0000 7f1bd781b640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:28:12.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:12.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:12.807 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:28:12.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:28:12.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:12.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:13.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:14.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:14.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:14.264 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:28:14.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:28:14.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:14.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:14.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:15.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:15.671 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:28:15.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:15.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:28:15.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:16.084 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:17.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:17.085 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:17.086 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:28:17.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:28:17.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:17.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:17.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:18.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:18.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:18.603 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:28:18.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:28:18.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:18.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:19.050 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:20.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:20.051 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:20.052 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:28:20.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:28:20.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:20.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:20.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:21.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:21.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:21.465 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:28:21.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:28:21.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:21.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:21.896 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:22.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:22.897 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:28:22.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:22.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:28:22.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:22.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:23.415 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:24.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:24.417 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:28:24.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:24.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:28:24.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:24.418 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:25.008 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:26.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:26.010 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:26.010 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:28:26.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:28:26.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:26.011 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:27.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:28.191 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:28.191 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:28:28.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:28.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:28:28.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:28.192 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:28.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:29.796 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:29.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:29.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:28:29.797 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:28:29.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:29.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:30.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:31.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:31.500 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:28:31.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:31.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:28:31.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:31.501 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:32.212 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:33.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:33.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:33.214 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:28:33.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:28:33.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:33.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:33.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:34.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:34.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:34.904 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:28:34.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:28:34.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:34.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:35.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:36.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:36.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:36.404 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:28:36.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:28:36.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:36.405 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:28:36.906 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3822375775,v1:127.0.0.1:6811/3822375775] [v2:127.0.0.1:6812/3822375775,v1:127.0.0.1:6813/3822375775] exists,up 2f81b04a-e228-4965-afbb-74d9b871fd65 2024-05-06T04:28:36.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:28:36.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:28:36.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:28:36.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-05-06T04:28:36.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:28:36.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e37a4b3-334e-449d-a184-e641dc315bcb 2024-05-06T04:28:36.913 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 0e37a4b3-334e-449d-a184-e641dc315bcb 2024-05-06T04:28:36.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0e37a4b3-334e-449d-a184-e641dc315bcb' 2024-05-06T04:28:36.914 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:28:36.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0XDhmkH8lNxAAG2upnEVZRrn2wkcj/DiCXw== 2024-05-06T04:28:36.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0XDhmkH8lNxAAG2upnEVZRrn2wkcj/DiCXw=="}' 2024-05-06T04:28:36.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e37a4b3-334e-449d-a184-e641dc315bcb -i td/test-snaptrim-stats/2/new.json 2024-05-06T04:28:37.549 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:28:37.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-05-06T04:28:37.561 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQB0XDhmkH8lNxAAG2upnEVZRrn2wkcj/DiCXw== --osd-uuid 0e37a4b3-334e-449d-a184-e641dc315bcb 2024-05-06T04:28:37.589 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.587+0000 7f593ae2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:37.598 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.597+0000 7f593ae2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:37.599 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.598+0000 7f593ae2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:37.601 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.599+0000 7f593ae2e640 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-05-06T04:28:37.601 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.599+0000 7f593ae2e640 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-05-06T04:28:37.601 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.599+0000 7f593ae2e640 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-05-06T04:28:37.665 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:37.664+0000 7f593ae2e640 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-05-06T04:28:43.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-05-06T04:28:43.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:28:43.676 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:28:43.676 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:28:43.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:28:44.225 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:28:44.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:28:44.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=30be1541-e865-4f5c-b51e-2c9f0f463167 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.172593/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-05-06T04:28:44.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:28:44.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:28:44.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:28:44.250 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:44.249+0000 7f842fe19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:44.259 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:44.258+0000 7f842fe19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:44.260 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:44.259+0000 7f842fe19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:28:44.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:28:44.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:28:44.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:28:44.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:28:44.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:28:44.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:44.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:28:44.674 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:28:44.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:44.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:44.794 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:44.793+0000 7f842fe19640 -1 Falling back to public interface 2024-05-06T04:28:45.125 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:46.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:46.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:46.127 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:28:46.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:28:46.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:46.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:46.551 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:47.231 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:28:47.230+0000 7f842fe19640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:28:47.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:47.553 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:47.553 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:28:47.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:28:47.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:47.554 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:48.027 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:49.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:49.028 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:49.029 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:28:49.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:28:49.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:49.029 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:49.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:50.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:50.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:50.465 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:28:50.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:28:50.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:50.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:50.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:51.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:51.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:51.874 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:28:51.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:28:51.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:51.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:52.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:53.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:53.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:53.346 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:28:53.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:28:53.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:53.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:53.751 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:54.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:54.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:54.754 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:28:54.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:28:54.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:54.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:55.213 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:56.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:56.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:56.215 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:28:56.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:28:56.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:56.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:56.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:57.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:57.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:57.618 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:28:57.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:28:57.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:57.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:58.194 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:28:59.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:28:59.195 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:28:59.195 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:28:59.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:28:59.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:28:59.196 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:28:59.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:29:00.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:29:00.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:29:00.618 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:29:00.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:29:00.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:29:00.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:29:01.595 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:29:02.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:29:02.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:29:02.597 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:29:02.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:29:02.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:29:02.598 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:29:03.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:29:04.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:29:04.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:29:04.816 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:29:04.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:29:04.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:29:04.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:29:05.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:29:06.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:29:06.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:29:06.626 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:29:06.626 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:29:06.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:29:06.630 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:29:07.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:29:08.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:29:08.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:29:08.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:29:08.439 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:29:08.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:29:08.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:29:09.446 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:29:10.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:29:10.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:29:10.448 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:29:10.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:29:10.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:29:10.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:29:10.904 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3578847458,v1:127.0.0.1:6819/3578847458] [v2:127.0.0.1:6820/3578847458,v1:127.0.0.1:6821/3578847458] exists,up 0e37a4b3-334e-449d-a184-e641dc315bcb 2024-05-06T04:29:10.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:29:10.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:29:10.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:29:10.905 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-05-06T04:29:11.808 INFO:tasks.workunit.client.0.smithi163.stderr:noscrub is set 2024-05-06T04:29:11.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-05-06T04:29:12.621 INFO:tasks.workunit.client.0.smithi163.stderr:nodeep-scrub is set 2024-05-06T04:29:12.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-05-06T04:29:12.632 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-05-06T04:29:13.803 INFO:tasks.workunit.client.0.smithi163.stderr:pool 'test' already exists 2024-05-06T04:29:13.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-06T04:29:14.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-05-06T04:29:14.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:29:14.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:29:14.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:29:14.818 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:29:14.819 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:29:14.819 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:29:14.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:29:14.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:29:14.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:29:14.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:29:14.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:29:14.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:29:14.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:29:14.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:29:14.911 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:29:15.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:29:15.326 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T04:29:15.326 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T04:29:15.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:29:15.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:15.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:29:15.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-05-06T04:29:15.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-05-06T04:29:15.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-05-06T04:29:15.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:15.439 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:29:15.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-05-06T04:29:15.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-05-06T04:29:15.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2024-05-06T04:29:15.560 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:15.560 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T04:29:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509446 2024-05-06T04:29:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509446 2024-05-06T04:29:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509446' 2024-05-06T04:29:15.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:15.672 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-05-06T04:29:15.673 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:15.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:29:15.673 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-05-06T04:29:15.674 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:15.674 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-05-06T04:29:15.674 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836499 2024-05-06T04:29:15.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-05-06T04:29:15.675 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:29:16.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836499 2024-05-06T04:29:16.065 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:16.065 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-05-06T04:29:16.065 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:16.066 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:29:16.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-05-06T04:29:16.067 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:16.067 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-05-06T04:29:16.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-05-06T04:29:16.068 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672977 2024-05-06T04:29:16.068 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:29:16.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-05-06T04:29:16.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:16.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509446 2024-05-06T04:29:16.465 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:16.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T04:29:16.466 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509446 2024-05-06T04:29:16.466 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:16.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509446 2024-05-06T04:29:16.467 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509446 2024-05-06T04:29:16.468 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-05-06T04:29:16.468 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T04:29:16.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509446 -lt 64424509446 2024-05-06T04:29:16.863 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:29:16.863 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:29:16.863 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:29:17.407 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:29:17.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-05-06T04:29:17.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:29:17.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:29:17.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:29:18.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-05-06T04:29:18.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:29:18.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:29:18.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-05-06T04:29:18.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-05-06T04:29:18.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-05-06T04:29:18.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-05-06T04:29:18.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-05-06T04:29:18.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-05-06T04:29:18.846 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records in 2024-05-06T04:29:18.846 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records out 2024-05-06T04:29:18.846 INFO:tasks.workunit.client.0.smithi163.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.2463e-05 s, 78.1 MB/s 2024-05-06T04:29:18.846 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-05-06T04:29:18.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:18.847 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-05-06T04:29:18.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:18.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-05-06T04:29:19.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:19.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-05-06T04:29:19.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:19.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-05-06T04:29:19.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:19.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-05-06T04:29:19.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:19.564 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-05-06T04:29:19.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:19.698 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-05-06T04:29:19.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:19.866 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-05-06T04:29:20.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:20.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-05-06T04:29:20.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:20.168 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-05-06T04:29:20.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-05-06T04:29:20.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-05-06T04:29:20.303 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-05-06T04:29:20.303 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-05-06T04:29:20.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-05-06T04:29:20.477 INFO:tasks.workunit.client.0.smithi163.stdout:created pool test snap snap1 2024-05-06T04:29:20.478 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-05-06T04:29:20.479 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-05-06T04:29:20.479 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records in 2024-05-06T04:29:20.479 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records out 2024-05-06T04:29:20.479 INFO:tasks.workunit.client.0.smithi163.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.024e-05 s, 81.5 MB/s 2024-05-06T04:29:20.480 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-05-06T04:29:20.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:20.480 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-05-06T04:29:20.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:20.797 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-05-06T04:29:20.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:20.956 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-05-06T04:29:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-05-06T04:29:21.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-05-06T04:29:21.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-05-06T04:29:21.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-05-06T04:29:21.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.677 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-05-06T04:29:21.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.811 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-05-06T04:29:21.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:21.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-05-06T04:29:22.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-05-06T04:29:22.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-05-06T04:29:22.097 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-05-06T04:29:22.246 INFO:tasks.workunit.client.0.smithi163.stdout:created pool test snap snap2 2024-05-06T04:29:22.248 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-05-06T04:29:22.249 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-05-06T04:29:22.249 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records in 2024-05-06T04:29:22.249 INFO:tasks.workunit.client.0.smithi163.stderr:1+0 records out 2024-05-06T04:29:22.249 INFO:tasks.workunit.client.0.smithi163.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.1315e-05 s, 79.8 MB/s 2024-05-06T04:29:22.250 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-05-06T04:29:22.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:22.250 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-05-06T04:29:22.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:22.591 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-05-06T04:29:22.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:22.725 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-05-06T04:29:22.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:22.868 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-05-06T04:29:23.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:23.044 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-05-06T04:29:23.178 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:23.179 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-05-06T04:29:23.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:23.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-05-06T04:29:23.463 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:23.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-05-06T04:29:23.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:23.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-05-06T04:29:23.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-05-06T04:29:23.757 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-05-06T04:29:23.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-05-06T04:29:23.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-05-06T04:29:23.909 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-05-06T04:29:23.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-05-06T04:29:23.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-05-06T04:29:24.108 INFO:tasks.workunit.client.0.smithi163.stdout:removed pool test snap snap1 2024-05-06T04:29:24.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-05-06T04:29:24.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:29:24.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:29:24.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:29:24.110 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:29:24.111 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:29:24.111 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:29:24.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:29:24.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:29:24.111 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:29:24.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:29:24.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:29:24.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:29:24.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:29:24.193 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:29:24.193 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:29:24.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:29:24.601 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T04:29:24.602 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T04:29:24.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:29:24.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:24.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:29:24.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-05-06T04:29:24.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-05-06T04:29:24.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-05-06T04:29:24.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:24.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:29:24.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-05-06T04:29:24.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-05-06T04:29:24.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672981' 2024-05-06T04:29:24.940 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:24.940 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T04:29:25.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-05-06T04:29:25.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-05-06T04:29:25.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509450' 2024-05-06T04:29:25.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:25.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-05-06T04:29:25.119 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:25.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:29:25.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-05-06T04:29:25.120 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:25.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-05-06T04:29:25.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-05-06T04:29:25.121 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836503 2024-05-06T04:29:25.121 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:29:25.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-05-06T04:29:25.526 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:29:26.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:29:26.528 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:29:26.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836503 2024-05-06T04:29:26.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:26.932 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-05-06T04:29:26.932 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:26.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:29:26.933 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-05-06T04:29:26.933 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:26.933 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-05-06T04:29:26.934 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672981 2024-05-06T04:29:26.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-05-06T04:29:26.934 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:29:27.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-05-06T04:29:27.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:27.375 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-05-06T04:29:27.375 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:27.375 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T04:29:27.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-05-06T04:29:27.376 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-05-06T04:29:27.377 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-05-06T04:29:27.377 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509450 2024-05-06T04:29:27.377 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T04:29:27.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-05-06T04:29:27.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:29:27.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:29:27.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:29:28.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-05-06T04:29:28.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:29:28.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:29:28.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:29:28.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:29:28.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:29:28.349 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:29:28.349 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:29:28.750 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-05-06T04:29:28.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:29:28.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:29:28.750 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:29:29.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-05-06T04:29:29.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:29:29.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:29:29.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-05-06T04:29:39.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-05-06T04:29:39.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-05-06T04:29:39.286 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-05-06T04:29:39.287 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-05-06T04:29:39.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:39.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-05-06T04:29:39.289 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-05-06T04:29:39.289 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:39.416 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-05-06T04:29:39.417 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-05-06T04:29:39.418 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:39.418 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-05-06T04:29:39.418 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:39.535 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.68861762999999998 2024-05-06T04:29:39.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.68861762999999998 2024-05-06T04:29:39.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:39.535 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-05-06T04:29:39.535 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-05-06T04:29:39.536 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:39.651 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-05-06T04:29:39.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-05-06T04:29:39.653 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:39.653 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-05-06T04:29:39.653 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:39.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .68861762999999998 + 0.47437070599999998 2024-05-06T04:29:39.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.16298833599999996 2024-05-06T04:29:39.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:39.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-05-06T04:29:39.771 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-05-06T04:29:39.772 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:39.889 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-05-06T04:29:39.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-05-06T04:29:39.890 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:39.890 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-05-06T04:29:39.891 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:40.008 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.16298833599999996 + 0.000321503 2024-05-06T04:29:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.16330983899999996 2024-05-06T04:29:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-05-06T04:29:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-05-06T04:29:40.009 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:40.129 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-05-06T04:29:40.130 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-05-06T04:29:40.130 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:40.130 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-05-06T04:29:40.130 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:40.246 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.16330983899999996 + 0.47071004900000002 2024-05-06T04:29:40.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.63401988799999998 2024-05-06T04:29:40.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:40.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-05-06T04:29:40.247 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-05-06T04:29:40.247 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:40.363 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-05-06T04:29:40.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-05-06T04:29:40.365 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:40.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-05-06T04:29:40.365 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:40.483 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.63401988799999998 + 0.35291839800000002 2024-05-06T04:29:40.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.98693828600000000 2024-05-06T04:29:40.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:40.483 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-05-06T04:29:40.484 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-05-06T04:29:40.484 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:40.601 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-05-06T04:29:40.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-05-06T04:29:40.602 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:40.602 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-05-06T04:29:40.602 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:40.719 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.98693828600000000 + 0.27751641599999999 2024-05-06T04:29:40.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.26445470199999999 2024-05-06T04:29:40.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:40.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-05-06T04:29:40.720 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-05-06T04:29:40.720 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:40.858 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-05-06T04:29:40.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-05-06T04:29:40.859 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:40.859 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-05-06T04:29:40.860 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:40.978 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.26445470199999999 + 0.83072521799999999 2024-05-06T04:29:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.09517991999999998 2024-05-06T04:29:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:40.978 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-05-06T04:29:40.979 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-05-06T04:29:40.979 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:41.095 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-05-06T04:29:41.096 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-05-06T04:29:41.097 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:41.097 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-05-06T04:29:41.098 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:41.214 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.09517991999999998 + 0.97439807000000001 2024-05-06T04:29:41.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.06957798999999999 2024-05-06T04:29:41.214 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-05-06T04:29:41.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.06957798999999999 > 0.0' 2024-05-06T04:29:41.215 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:41.215 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:29:41.215 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-05-06T04:29:41.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-05-06T04:29:41.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-05-06T04:29:41.216 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-05-06T04:29:41.376 INFO:tasks.workunit.client.0.smithi163.stdout:removed pool test snap snap2 2024-05-06T04:29:41.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-05-06T04:29:41.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-06T04:29:41.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-06T04:29:41.379 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-06T04:29:41.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-06T04:29:41.380 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-06T04:29:41.380 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-06T04:29:41.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-06T04:29:41.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-06T04:29:41.380 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-06T04:29:41.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-06T04:29:41.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-06T04:29:41.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-06T04:29:41.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-06T04:29:41.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-05-06T04:29:41.462 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-05-06T04:29:41.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-05-06T04:29:41.858 INFO:tasks.workunit.client.0.smithi163.stderr:1 2024-05-06T04:29:41.858 INFO:tasks.workunit.client.0.smithi163.stderr:2' 2024-05-06T04:29:41.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-05-06T04:29:41.858 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:41.859 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-06T04:29:42.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-05-06T04:29:42.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-05-06T04:29:42.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-05-06T04:29:42.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:42.127 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-06T04:29:42.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672987 2024-05-06T04:29:42.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672987 2024-05-06T04:29:42.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672987' 2024-05-06T04:29:42.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-05-06T04:29:42.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-06T04:29:42.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-05-06T04:29:42.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-05-06T04:29:42.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-64424509456' 2024-05-06T04:29:42.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:42.455 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-05-06T04:29:42.455 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:42.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-05-06T04:29:42.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-05-06T04:29:42.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:42.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-05-06T04:29:42.458 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.0 seq 21474836509 2024-05-06T04:29:42.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-05-06T04:29:42.458 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:29:42.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836509 2024-05-06T04:29:42.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-05-06T04:29:43.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-06T04:29:43.883 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-06T04:29:44.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-05-06T04:29:44.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:44.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672987 2024-05-06T04:29:44.325 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:44.326 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-05-06T04:29:44.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672987 2024-05-06T04:29:44.326 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:44.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672987 2024-05-06T04:29:44.327 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-05-06T04:29:44.328 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.1 seq 42949672987 2024-05-06T04:29:44.328 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-06T04:29:44.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672987 -lt 42949672987 2024-05-06T04:29:44.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-05-06T04:29:44.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-05-06T04:29:44.720 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-05-06T04:29:44.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-05-06T04:29:44.721 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-05-06T04:29:44.721 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-05-06T04:29:44.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-05-06T04:29:44.722 INFO:tasks.workunit.client.0.smithi163.stdout:waiting osd.2 seq 64424509456 2024-05-06T04:29:44.722 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-05-06T04:29:44.722 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-06T04:29:45.136 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509456 2024-05-06T04:29:45.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-06T04:29:45.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:29:45.136 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:29:45.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-05-06T04:29:45.680 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-06T04:29:45.680 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-06T04:29:45.680 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-06T04:29:45.680 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-06T04:29:45.680 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-06T04:29:45.681 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-06T04:29:45.681 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-06T04:29:46.083 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-05-06T04:29:46.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-06T04:29:46.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-06T04:29:46.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-06T04:29:46.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-05-06T04:29:46.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-06T04:29:46.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-06T04:29:46.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-05-06T04:29:56.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-05-06T04:29:56.614 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-05-06T04:29:56.614 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-05-06T04:29:56.615 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-05-06T04:29:56.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:56.616 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-05-06T04:29:56.617 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-05-06T04:29:56.617 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:56.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-05-06T04:29:56.746 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-05-06T04:29:56.746 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:56.747 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-05-06T04:29:56.747 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:56.876 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 1.020236143 2024-05-06T04:29:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.020236143 2024-05-06T04:29:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-05-06T04:29:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-05-06T04:29:56.877 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:56.995 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-05-06T04:29:56.995 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-05-06T04:29:56.996 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:56.997 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-05-06T04:29:56.997 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:57.114 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.020236143 + 0.41963028499999999 2024-05-06T04:29:57.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.43986642799999999 2024-05-06T04:29:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-05-06T04:29:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-05-06T04:29:57.115 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:57.232 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-05-06T04:29:57.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-05-06T04:29:57.233 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:57.233 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-05-06T04:29:57.234 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:57.351 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.43986642799999999 + 0.00041544800000000001 2024-05-06T04:29:57.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.44028187599999999001 2024-05-06T04:29:57.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:57.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-05-06T04:29:57.352 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-05-06T04:29:57.352 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:57.468 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-05-06T04:29:57.469 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-05-06T04:29:57.470 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:57.470 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-05-06T04:29:57.470 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:57.585 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.44028187599999999001 + 0.52892314299999998 2024-05-06T04:29:57.585 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.96920501899999997001 2024-05-06T04:29:57.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:57.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-05-06T04:29:57.586 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-05-06T04:29:57.586 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:57.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-05-06T04:29:57.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-05-06T04:29:57.704 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:57.704 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-05-06T04:29:57.704 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:57.821 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.96920501899999997001 + 0.87285155299999995 2024-05-06T04:29:57.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.84205657199999992001 2024-05-06T04:29:57.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:57.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-05-06T04:29:57.822 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-05-06T04:29:57.822 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:57.951 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-05-06T04:29:57.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-05-06T04:29:57.952 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:57.953 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-05-06T04:29:57.953 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:58.069 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.84205657199999992001 + 0.26414676100000001 2024-05-06T04:29:58.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.10620333299999993001 2024-05-06T04:29:58.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:58.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-05-06T04:29:58.071 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-05-06T04:29:58.071 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:58.187 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-05-06T04:29:58.188 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-05-06T04:29:58.189 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:58.189 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-05-06T04:29:58.189 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:58.306 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.10620333299999993001 + 0.63296213199999996 2024-05-06T04:29:58.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.73916546499999989001 2024-05-06T04:29:58.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-05-06T04:29:58.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-05-06T04:29:58.308 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-05-06T04:29:58.308 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-05-06T04:29:58.424 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-05-06T04:29:58.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-05-06T04:29:58.426 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:58.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-05-06T04:29:58.426 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-05-06T04:29:58.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.73916546499999989001 + 0.63287969499999996 2024-05-06T04:29:58.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.37204515999999985001 2024-05-06T04:29:58.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-05-06T04:29:58.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.37204515999999985001 > 0.0' 2024-05-06T04:29:58.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-05-06T04:29:58.543 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:29:58.544 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-05-06T04:29:58.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-05-06T04:29:58.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-05-06T04:29:58.545 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:29:58.546 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:29:58.652 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:29:58.652 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:29:58.653 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:29:58.653 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:29:58.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:29:58.654 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:29:58.655 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:29:58.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.655 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:29:58.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:29:58.656 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:29:58.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:29:58.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:29:58.687 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:29:58.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.688 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:29:58.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:29:58.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:29:58.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:29:58.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-05-06T04:29:58.689 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:29:58.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:29:58.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:29:58.691 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:29:58.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:29:58.692 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:29:58.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:29:58.693 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:29:58.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:29:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:29:58.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:29:58.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:29:58.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:29:58.696 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:29:58.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:29:58.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.697 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:29:58.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/test-snaptrim-stats 0 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:29:58.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:29:58.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:29:58.700 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:29:58.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:29:58.701 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:29:58.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:29:58.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:29:58.702 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:29:58.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:29:58.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:29:58.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:29:58.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-06T04:29:58.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-05-06T04:29:58.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:29:58.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.706 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.172593 2024-05-06T04:29:58.706 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.172593 2024-05-06T04:29:58.707 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:29:58.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:29:58.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-05-06T04:29:58.708 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-06T04:29:58.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T04:29:58.731 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-05-06T04:29:58.731 DEBUG:teuthology.orchestra.run.smithi163:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-05-06T04:29:58.791 INFO:tasks.workunit.client.0.smithi163.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-06T04:29:58.801 INFO:tasks.workunit.client.0.smithi163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ver-health 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-05-06T04:29:58.802 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none ' 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-05-06T04:29:58.803 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_check_version_health_2 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:TEST_check_version_health_3' 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:29:58.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:29:58.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:29:58.805 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:29:58.805 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:29:58.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:29:58.806 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:29:58.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:29:58.806 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:29:58.807 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-05-06T04:29:58.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:29:58.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.808 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:58.808 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.187931 2024-05-06T04:29:58.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:29:58.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:29:58.809 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-05-06T04:29:58.809 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:29:58.810 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.810 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:58.810 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.187931 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-05-06T04:29:58.812 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:29:58.813 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:29:58.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:29:58.814 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:29:58.814 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:29:58.814 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:29:58.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:29:58.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:29:58.815 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:29:58.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:29:58.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:29:58.816 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:29:58.816 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:29:58.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:29:58.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-05-06T04:29:58.818 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:29:58.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.819 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:58.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.187931 2024-05-06T04:29:58.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:29:58.819 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:29:58.820 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-05-06T04:29:58.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:29:58.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:58.820 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:58.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.187931 2024-05-06T04:29:58.822 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:29:58.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:29:58.822 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-05-06T04:29:58.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-05-06T04:29:59.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:29:59.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:29:59.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:29:59.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:29:59.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:59.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:59.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:29:59.284 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-05-06T04:29:59.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:29:59.477 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:29:59.478 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:59.478 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:59.478 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.a.asok 2024-05-06T04:29:59.478 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:29:59.478 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.a.asok config get fsid 2024-05-06T04:29:59.558 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:29:59.559 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.a.asok 2024-05-06T04:29:59.560 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:29:59.560 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.a.asok config get mon_host 2024-05-06T04:29:59.637 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-05-06T04:29:59.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-05-06T04:29:59.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:29:59.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T04:29:59.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:29:59.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-05-06T04:29:59.638 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-05-06T04:30:00.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:30:00.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:30:00.492 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:30:00.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:30:00.492 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:30:00.493 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:30:00.493 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:30:00.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-05-06T04:30:00.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:30:00.693 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:30:00.694 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:30:00.694 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.b.asok 2024-05-06T04:30:00.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:30:00.694 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.b.asok config get fsid 2024-05-06T04:30:00.777 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:30:00.778 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.b.asok 2024-05-06T04:30:00.779 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:30:00.779 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.b.asok config get mon_host 2024-05-06T04:30:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-05-06T04:30:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-05-06T04:30:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:30:00.859 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:30:00.860 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:30:00.861 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:30:00.862 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 56ad0819-0a9a-49df-b73f-993835489ed5 2024-05-06T04:30:00.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56ad0819-0a9a-49df-b73f-993835489ed5 2024-05-06T04:30:00.863 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 56ad0819-0a9a-49df-b73f-993835489ed5' 2024-05-06T04:30:00.863 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:30:00.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIXDhmALMwNBAA14wU61aBRnOm2SIRqjL3ag== 2024-05-06T04:30:00.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIXDhmALMwNBAA14wU61aBRnOm2SIRqjL3ag=="}' 2024-05-06T04:30:00.877 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56ad0819-0a9a-49df-b73f-993835489ed5 -i td/ver-health/0/new.json 2024-05-06T04:30:07.207 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:30:07.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-05-06T04:30:07.218 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIXDhmALMwNBAA14wU61aBRnOm2SIRqjL3ag== --osd-uuid 56ad0819-0a9a-49df-b73f-993835489ed5 2024-05-06T04:30:07.246 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.245+0000 7f146f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.247+0000 7f146f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:07.250 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.249+0000 7f146f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:07.251 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.249+0000 7f146f80c640 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-05-06T04:30:07.251 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.249+0000 7f146f80c640 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-05-06T04:30:07.251 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.249+0000 7f146f80c640 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-05-06T04:30:07.363 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:07.362+0000 7f146f80c640 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-05-06T04:30:13.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-05-06T04:30:13.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:30:13.082 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:30:13.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:30:13.082 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:30:13.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:30:13.286 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:30:13.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:30:13.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:30:13.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:30:13.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:30:13.312 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:13.310+0000 7f6d98933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:13.318 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:13.317+0000 7f6d98933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:13.319 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:13.318+0000 7f6d98933640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:13.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:30:13.512 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:30:13.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:30:13.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:30:13.513 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:30:13.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:30:13.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:13.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:30:13.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:13.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:13.740 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:13.853 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:13.852+0000 7f6d98933640 -1 Falling back to public interface 2024-05-06T04:30:14.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:14.741 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:14.741 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:30:14.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:30:14.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:14.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:14.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:15.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:15.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:15.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:30:15.955 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:30:15.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:15.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:16.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:16.156 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:16.154+0000 7f6d98933640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:30:17.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:17.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:17.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:30:17.150 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:30:17.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:17.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:17.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:18.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:18.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:18.705 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:30:18.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:30:18.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:18.705 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:18.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:19.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:19.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:19.903 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:30:19.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:30:19.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:19.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:20.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:21.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:21.102 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:21.102 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:30:21.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:30:21.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:21.103 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:21.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:22.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:22.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:22.294 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:30:22.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:30:22.294 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:22.295 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:22.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:23.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:23.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:23.493 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:30:23.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:30:23.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:23.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:23.690 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:24.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:24.691 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:24.692 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:30:24.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:30:24.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:24.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:24.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:25.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:25.890 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:25.890 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:30:25.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:30:25.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:25.891 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:26.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:27.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:27.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:27.088 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:30:27.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:30:27.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:27.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:27.285 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:28.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:28.286 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:28.287 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:30:28.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:30:28.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:28.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:28.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:29.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:29.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:29.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:30:29.486 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:30:29.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:29.486 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:29.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:30.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:30.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:30.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:30:30.684 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:30:30.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:30.684 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:30.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:31.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:31.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:31.879 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:30:31.879 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:30:31.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:31.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:32.074 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:33.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:33.075 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:33.076 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:30:33.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:30:33.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:33.076 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:33.268 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:34.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:34.270 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:34.270 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:30:34.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:30:34.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:34.271 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:34.462 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:35.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:35.464 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:35.464 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:30:35.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:30:35.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:35.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:35.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:36.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:36.671 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:36.671 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:30:36.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:30:36.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:36.672 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:37.089 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:38.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:38.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:38.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:30:38.091 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:30:38.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:38.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:38.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:39.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:39.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:39.291 INFO:tasks.workunit.client.0.smithi163.stdout:21 2024-05-06T04:30:39.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-06T04:30:39.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:39.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:30:39.532 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/211357888,v1:127.0.0.1:6801/211357888] [v2:127.0.0.1:6802/211357888,v1:127.0.0.1:6803/211357888] exists,up 56ad0819-0a9a-49df-b73f-993835489ed5 2024-05-06T04:30:39.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:30:39.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:30:39.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:30:39.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-05-06T04:30:39.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:30:39.538 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:30:39.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56ff42db-9978-4878-abc6-d46a0881da05 2024-05-06T04:30:39.540 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 56ff42db-9978-4878-abc6-d46a0881da05 2024-05-06T04:30:39.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 56ff42db-9978-4878-abc6-d46a0881da05' 2024-05-06T04:30:39.541 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:30:39.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvXDhm4IbZIBAA4q7jJ+MzVYeAMSc5s6XUAw== 2024-05-06T04:30:39.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvXDhm4IbZIBAA4q7jJ+MzVYeAMSc5s6XUAw=="}' 2024-05-06T04:30:39.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56ff42db-9978-4878-abc6-d46a0881da05 -i td/ver-health/1/new.json 2024-05-06T04:30:40.025 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:30:40.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-05-06T04:30:40.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvXDhm4IbZIBAA4q7jJ+MzVYeAMSc5s6XUAw== --osd-uuid 56ff42db-9978-4878-abc6-d46a0881da05 2024-05-06T04:30:40.062 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.061+0000 7f5686c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:40.064 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.063+0000 7f5686c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:40.065 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.064+0000 7f5686c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:40.065 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.064+0000 7f5686c11640 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-05-06T04:30:40.065 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.064+0000 7f5686c11640 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-05-06T04:30:40.066 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.064+0000 7f5686c11640 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-05-06T04:30:40.177 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:40.176+0000 7f5686c11640 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-05-06T04:30:46.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-05-06T04:30:46.156 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:30:46.157 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:30:46.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:30:46.157 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:30:46.362 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:30:46.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:30:46.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:30:46.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:30:46.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:30:46.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:30:46.388 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:46.386+0000 7fb271224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:46.392 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:46.390+0000 7fb271224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:46.393 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:46.391+0000 7fb271224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:30:46.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:30:46.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:30:46.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:30:46.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:30:46.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:30:46.600 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:30:46.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:46.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:30:46.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:46.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:46.837 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:46.926 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:46.924+0000 7fb271224640 -1 Falling back to public interface 2024-05-06T04:30:47.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:47.839 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:47.839 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:30:47.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:30:47.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:47.840 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:48.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:49.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:49.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:49.037 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:30:49.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:30:49.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:49.037 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:49.226 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:49.258 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:30:49.256+0000 7fb271224640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:30:50.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:50.228 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:50.228 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:30:50.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:30:50.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:50.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:50.575 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:51.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:51.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:51.576 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:30:51.577 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:30:51.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:51.577 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:51.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:52.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:52.776 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:52.776 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:30:52.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:30:52.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:52.777 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:52.973 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:53.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:53.975 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:53.975 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:30:53.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:30:53.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:53.976 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:54.174 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:55.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:55.175 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:55.176 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:30:55.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:30:55.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:55.176 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:55.372 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:56.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:56.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:56.373 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:30:56.373 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:30:56.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:56.374 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:56.594 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:57.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:57.596 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:57.596 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:30:57.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:30:57.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:57.597 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:57.793 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:58.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:58.794 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:58.795 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:30:58.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:30:58.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:58.795 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:30:58.990 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:30:59.991 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:30:59.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:30:59.992 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:30:59.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:30:59.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:30:59.993 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:00.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:01.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:01.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:01.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:31:01.190 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:31:01.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:01.190 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:01.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:02.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:02.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:02.389 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:31:02.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:31:02.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:02.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:02.586 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:03.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:03.588 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:03.588 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:31:03.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:31:03.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:03.589 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:03.784 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:04.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:04.786 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:04.786 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:31:04.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:31:04.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:04.787 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:04.984 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:05.985 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:05.986 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:05.986 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:31:05.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:31:05.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:05.987 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:06.184 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:07.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:07.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:07.185 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:31:07.186 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:31:07.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:07.186 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:07.458 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:08.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:08.460 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:08.460 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:31:08.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:31:08.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:08.461 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:08.656 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:09.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:09.658 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:09.658 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:31:09.659 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:31:09.659 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:09.659 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:09.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:10.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:10.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:10.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:31:10.854 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:31:10.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:10.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:11.049 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:12.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:12.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:12.052 INFO:tasks.workunit.client.0.smithi163.stdout:21 2024-05-06T04:31:12.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-06T04:31:12.052 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:12.057 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:12.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:13.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:13.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:13.243 INFO:tasks.workunit.client.0.smithi163.stdout:22 2024-05-06T04:31:13.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-05-06T04:31:13.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:13.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:13.465 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:14.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:14.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:14.466 INFO:tasks.workunit.client.0.smithi163.stdout:23 2024-05-06T04:31:14.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-05-06T04:31:14.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:14.467 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:14.697 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2943419463,v1:127.0.0.1:6809/2943419463] [v2:127.0.0.1:6810/2943419463,v1:127.0.0.1:6811/2943419463] exists,up 56ff42db-9978-4878-abc6-d46a0881da05 2024-05-06T04:31:14.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:31:14.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-05-06T04:31:14.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:31:14.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=129a535c-965f-4dd6-aa97-355ff31be716 2024-05-06T04:31:14.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 129a535c-965f-4dd6-aa97-355ff31be716' 2024-05-06T04:31:14.705 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 129a535c-965f-4dd6-aa97-355ff31be716 2024-05-06T04:31:14.705 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:31:14.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASXThmEQG+KhAABGvHpWkJ3zp4XNCjQrNKxQ== 2024-05-06T04:31:14.718 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASXThmEQG+KhAABGvHpWkJ3zp4XNCjQrNKxQ=="}' 2024-05-06T04:31:14.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 129a535c-965f-4dd6-aa97-355ff31be716 -i td/ver-health/2/new.json 2024-05-06T04:31:15.262 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:31:15.272 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-05-06T04:31:15.273 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASXThmEQG+KhAABGvHpWkJ3zp4XNCjQrNKxQ== --osd-uuid 129a535c-965f-4dd6-aa97-355ff31be716 2024-05-06T04:31:15.301 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.299+0000 7f27b60a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:15.302 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.301+0000 7f27b60a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:15.304 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.302+0000 7f27b60a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:15.304 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.303+0000 7f27b60a6640 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-05-06T04:31:15.304 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.303+0000 7f27b60a6640 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-05-06T04:31:15.304 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.303+0000 7f27b60a6640 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-05-06T04:31:15.513 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:15.512+0000 7f27b60a6640 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-05-06T04:31:21.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-05-06T04:31:21.241 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:31:21.242 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:31:21.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:31:21.242 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:31:21.447 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:31:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:31:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:31:21.448 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:31:21.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:31:21.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:31:21.474 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:21.473+0000 7fe0ad814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:21.476 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:21.475+0000 7fe0ad814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:21.477 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:21.476+0000 7fe0ad814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:21.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:31:21.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:31:21.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:31:21.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:31:21.685 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:31:21.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:21.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:31:21.686 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:31:21.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:21.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:21.873 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:22.001 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:22.000+0000 7fe0ad814640 -1 Falling back to public interface 2024-05-06T04:31:22.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:22.874 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:22.875 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:31:22.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:31:22.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:22.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:23.068 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:24.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:24.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:24.070 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:31:24.070 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:31:24.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:24.071 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:24.262 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:24.366 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:24.365+0000 7fe0ad814640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:31:25.263 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:25.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:25.264 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:31:25.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:31:25.264 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:25.265 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:25.474 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:26.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:26.475 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:26.476 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:31:26.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:31:26.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:26.476 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:26.669 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:27.670 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:27.673 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:27.674 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:31:27.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:31:27.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:27.675 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:27.952 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:28.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:28.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:28.953 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:31:28.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:31:28.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:28.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:29.151 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:30.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:30.153 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:30.153 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:31:30.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:31:30.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:30.154 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:30.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:31.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:31.351 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:31:31.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:31.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:31:31.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:31.352 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:31.547 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:32.549 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:32.550 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:32.550 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:31:32.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:31:32.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:32.552 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:32.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:33.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:33.747 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:33.747 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:31:33.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:31:33.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:33.748 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:33.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:34.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:34.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:34.945 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:31:34.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:31:34.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:34.945 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:35.140 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:36.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:36.142 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:36.142 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:31:36.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:31:36.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:36.143 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:36.336 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:37.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:37.337 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:37.337 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:31:37.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:31:37.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:37.338 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:37.536 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:38.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:38.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:38.537 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:31:38.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:31:38.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:38.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:38.735 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:39.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:39.737 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:39.737 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:31:39.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:31:39.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:39.738 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:39.931 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:40.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:40.932 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:40.933 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:31:40.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:31:40.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:40.934 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:41.131 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:42.132 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:42.133 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:42.133 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:31:42.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:31:42.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:42.134 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:42.331 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:43.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:43.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:43.332 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:31:43.333 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:31:43.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:43.333 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:43.527 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:44.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:44.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:44.529 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:31:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:31:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:44.724 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:45.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:45.726 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:31:45.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:31:45.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:45.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:45.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:46.923 INFO:tasks.workunit.client.0.smithi163.stdout:21 2024-05-06T04:31:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-06T04:31:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:46.924 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:47.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:48.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:48.121 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:48.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-05-06T04:31:48.122 INFO:tasks.workunit.client.0.smithi163.stdout:22 2024-05-06T04:31:48.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:48.122 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:31:48.315 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1109849088,v1:127.0.0.1:6817/1109849088] [v2:127.0.0.1:6818/1109849088,v1:127.0.0.1:6819/1109849088] exists,up 129a535c-965f-4dd6-aa97-355ff31be716 2024-05-06T04:31:48.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:31:48.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:31:48.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:31:48.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-05-06T04:31:53.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-05-06T04:31:53.510 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_OK 2024-05-06T04:31:53.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-05-06T04:31:53.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-05-06T04:31:53.744 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-05-06T04:31:53.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:31:53.745 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:31:53.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:31:53.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:31:53.745 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:31:53.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-05-06T04:31:53.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:31:53.854 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:31:53.855 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-05-06T04:31:53.856 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:31:53.857 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:31:53.877 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:53.876+0000 7f00cfa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:53.879 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:53.878+0000 7f00cfa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:53.880 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:53.879+0000 7f00cfa2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:31:54.291 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:31:54.292 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:31:54.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:54.293 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:54.395 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:54.394+0000 7f00cfa2e640 -1 Falling back to public interface 2024-05-06T04:31:54.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:55.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:55.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:55.518 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:31:55.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:31:55.518 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:55.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:55.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:56.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:56.714 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:56.714 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:31:56.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:31:56.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:56.715 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:31:57.208 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:31:57.207+0000 7f00cfa2e640 -1 osd.1 16 log_to_monitors true 2024-05-06T04:31:57.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:31:57.903 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:31:57.904 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:31:57.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:31:57.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:31:57.904 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:31:58.321 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6808/153759947,v1:127.0.0.1:6809/153759947] [v2:127.0.0.1:6810/153759947,v1:127.0.0.1:6811/153759947] exists,up 56ff42db-9978-4878-abc6-d46a0881da05 2024-05-06T04:31:58.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:31:58.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:31:58.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:31:58.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:31:58.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:31:58.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:31:58.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:31:59.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:31:59.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:31:59.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:31:59.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:31:59.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:00.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:00.914 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:00.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:00.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:32:01.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:32:02.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:03.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:03.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:03.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:03.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:32:03.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:04.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:04.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:04.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:04.517 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:32:04.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:05.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:05.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:05.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:05.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:32:05.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-05-06T04:32:05.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-05-06T04:32:05.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-05-06T04:32:05.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-05-06T04:32:05.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-05-06T04:32:05.992 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:06.180 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-05-06T04:32:06.180 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-05-06T04:32:06.180 INFO:tasks.workunit.client.0.smithi163.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-05-06T04:32:06.180 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] MGR_DOWN: no active mgr 2024-05-06T04:32:06.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:06.189 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-05-06T04:32:06.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:06.466 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-05-06T04:32:06.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:06.742 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-05-06T04:32:07.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-05-06T04:32:07.035 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:32:07.036 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:32:07.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:32:07.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:32:07.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:32:07.144 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:07.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-06T04:32:07.146 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:32:07.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:32:07.148 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-05-06T04:32:07.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-06T04:32:07.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:32:07.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:32:07.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:32:07.168 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:07.167+0000 7fe2e2014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:07.247 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:07.246+0000 7fe2e2014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:07.248 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:07.247+0000 7fe2e2014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:07.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:32:07.455 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:32:07.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:07.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:32:07.764 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:07.763+0000 7fe2e2014640 -1 Falling back to public interface 2024-05-06T04:32:07.851 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:08.852 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:08.852 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:32:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:32:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:08.853 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:32:09.039 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:10.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:10.040 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:10.041 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:32:10.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:32:10.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:10.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:32:10.229 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:10.496 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:10.495+0000 7fe2e2014640 -1 osd.2 19 log_to_monitors true 2024-05-06T04:32:11.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:11.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:11.231 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:32:11.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:32:11.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:11.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:32:11.428 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6816/2686811081,v1:127.0.0.1:6817/2686811081] [v2:127.0.0.1:6818/2686811081,v1:127.0.0.1:6819/2686811081] exists,up 129a535c-965f-4dd6-aa97-355ff31be716 2024-05-06T04:32:11.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:32:11.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:32:11.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:32:11.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-05-06T04:32:11.429 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:32:11.430 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:32:11.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:32:11.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:32:11.430 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:32:11.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:32:11.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-05-06T04:32:11.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-05-06T04:32:11.537 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-05-06T04:32:11.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-06T04:32:11.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-05-06T04:32:11.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-06T04:32:11.538 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-05-06T04:32:11.539 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:32:11.540 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-05-06T04:32:11.541 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:32:11.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:32:11.542 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-05-06T04:32:11.542 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-05-06T04:32:11.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-06T04:32:11.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-06T04:32:11.543 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-06T04:32:11.562 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:11.559+0000 7fb59a212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:11.563 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:11.561+0000 7fb59a212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:11.564 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:11.562+0000 7fb59a212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:12.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-05-06T04:32:12.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:32:12.001 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:32:12.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:32:12.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:32:12.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:12.002 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:32:12.002 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:32:12.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:12.003 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:12.085 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:12.083+0000 7fb59a212640 -1 Falling back to public interface 2024-05-06T04:32:12.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:13.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:13.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:13.256 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:32:13.256 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:32:13.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:13.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:13.449 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:14.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:14.450 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:14.451 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:32:14.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:32:14.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:14.451 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:14.642 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:14.670 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:14.669+0000 7fb59a212640 -1 osd.0 22 log_to_monitors true 2024-05-06T04:32:15.643 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:15.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:15.644 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:32:15.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:32:15.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:15.644 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:15.913 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/3052944523,v1:127.0.0.1:6801/3052944523] [v2:127.0.0.1:6802/3052944523,v1:127.0.0.1:6803/3052944523] exists,up 56ad0819-0a9a-49df-b73f-993835489ed5 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-05-06T04:32:15.915 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:15.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:15.916 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:16.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:17.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:17.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:17.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:17.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:17.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:18.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:18.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:18.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:18.368 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:18.570 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:19.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:19.571 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:19.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:19.572 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:19.772 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:32:20.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:32:20.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:32:20.774 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:32:20.778 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:21.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-05-06T04:32:21.022 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-05-06T04:32:21.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-05-06T04:32:21.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-05-06T04:32:21.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-05-06T04:32:21.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:21.234 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-05-06T04:32:21.235 INFO:tasks.workunit.client.0.smithi163.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-05-06T04:32:21.235 INFO:tasks.workunit.client.0.smithi163.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-05-06T04:32:21.235 INFO:tasks.workunit.client.0.smithi163.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-05-06T04:32:21.235 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] MGR_DOWN: no active mgr 2024-05-06T04:32:21.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:21.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-05-06T04:32:21.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:21.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-05-06T04:32:21.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:21.804 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-05-06T04:32:22.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-05-06T04:32:22.054 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-05-06T04:32:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-05-06T04:32:22.253 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:32:22.254 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:32:22.360 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:32:22.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:32:22.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:32:22.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:32:22.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:32:22.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:32:22.362 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:32:22.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:32:22.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:32:22.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:32:22.364 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:32:22.364 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:32:22.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:32:22.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-05-06T04:32:22.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:32:22.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:22.381 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:22.381 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.187931 2024-05-06T04:32:22.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:32:22.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:32:22.383 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:32:22.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:32:22.386 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:32:22.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:32:22.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:32:22.387 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:32:22.388 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:32:22.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:32:22.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:32:22.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-05-06T04:32:22.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:32:22.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:22.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:22.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.187931 2024-05-06T04:32:22.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:32:22.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:32:22.392 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-05-06T04:32:22.392 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:32:22.392 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:22.392 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:22.393 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.187931 2024-05-06T04:32:22.395 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:32:22.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:32:22.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:32:22.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-05-06T04:32:22.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-05-06T04:32:22.395 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:32:22.396 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:32:22.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:32:22.397 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:32:22.397 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:32:22.397 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:32:22.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:32:22.398 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:32:22.398 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:32:22.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:32:22.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:32:22.399 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:32:22.399 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:32:22.400 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:32:22.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-06T04:32:22.401 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-05-06T04:32:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:32:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:22.402 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:22.402 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.187931 2024-05-06T04:32:22.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:32:22.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:32:22.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-05-06T04:32:22.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-06T04:32:22.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:22.404 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:22.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.187931 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-06T04:32:22.406 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:32:22.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-05-06T04:32:22.407 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:22.872 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-05-06T04:32:23.081 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:32:23.081 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-06T04:32:23.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:32:23.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:32:23.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:32:23.082 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:23.083 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.a.asok 2024-05-06T04:32:23.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:32:23.084 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.a.asok config get fsid 2024-05-06T04:32:23.166 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.a.asok 2024-05-06T04:32:23.167 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:32:23.168 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.a.asok config get mon_host 2024-05-06T04:32:23.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-05-06T04:32:23.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-05-06T04:32:23.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:32:23.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T04:32:23.246 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:32:23.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-05-06T04:32:23.247 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-05-06T04:32:24.078 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:24.079 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-05-06T04:32:24.280 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:32:24.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T04:32:24.281 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:32:24.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T04:32:24.282 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:24.283 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.b.asok 2024-05-06T04:32:24.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:32:24.284 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.b.asok config get fsid 2024-05-06T04:32:24.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T04:32:24.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:32:24.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T04:32:24.370 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.b.asok 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:32:24.371 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.b.asok config get mon_host 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:32:24.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:32:24.454 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:32:24.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd1f0964-6e30-4b81-8960-e551aac65685 2024-05-06T04:32:24.456 INFO:tasks.workunit.client.0.smithi163.stdout:add osd0 cd1f0964-6e30-4b81-8960-e551aac65685 2024-05-06T04:32:24.457 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cd1f0964-6e30-4b81-8960-e551aac65685' 2024-05-06T04:32:24.457 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:32:24.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYXThmYE8BHBAArcqFewJ/SjZeIJR6p6rDBw== 2024-05-06T04:32:24.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYXThmYE8BHBAArcqFewJ/SjZeIJR6p6rDBw=="}' 2024-05-06T04:32:24.471 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd1f0964-6e30-4b81-8960-e551aac65685 -i td/ver-health/0/new.json 2024-05-06T04:32:30.814 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:32:30.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-05-06T04:32:30.826 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYXThmYE8BHBAArcqFewJ/SjZeIJR6p6rDBw== --osd-uuid cd1f0964-6e30-4b81-8960-e551aac65685 2024-05-06T04:32:30.853 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.852+0000 7ff0a97b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:30.855 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.854+0000 7ff0a97b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:30.856 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.855+0000 7ff0a97b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:30.856 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.855+0000 7ff0a97b4640 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-05-06T04:32:30.856 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.855+0000 7ff0a97b4640 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-05-06T04:32:30.856 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.855+0000 7ff0a97b4640 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-05-06T04:32:30.948 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:30.947+0000 7ff0a97b4640 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-05-06T04:32:36.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-05-06T04:32:36.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:32:36.423 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd0 key to auth repository 2024-05-06T04:32:36.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-06T04:32:36.424 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:32:36.627 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.0 2024-05-06T04:32:36.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-06T04:32:36.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:32:36.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:32:36.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:32:36.628 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:32:36.653 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:36.652+0000 7fdc05b99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:36.662 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:36.661+0000 7fdc05b99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:36.663 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:36.662+0000 7fdc05b99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:32:36.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-06T04:32:36.848 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:32:36.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-06T04:32:36.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:32:36.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:32:36.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:36.849 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:32:36.849 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:32:36.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:36.850 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:37.061 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:37.195 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:37.194+0000 7fdc05b99640 -1 Falling back to public interface 2024-05-06T04:32:38.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:38.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:38.062 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:32:38.063 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:32:38.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:38.063 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:38.257 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:39.258 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:39.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:39.259 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:32:39.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:32:39.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:39.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:39.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:39.642 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:32:39.641+0000 7fdc05b99640 -1 osd.0 0 log_to_monitors true 2024-05-06T04:32:40.454 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:40.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:40.455 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:32:40.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:32:40.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:40.456 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:40.726 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:41.727 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:41.727 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:32:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:32:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:41.728 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:41.925 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:42.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:42.926 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:42.926 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:32:42.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:32:42.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:42.927 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:43.126 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:44.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:44.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:44.127 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:32:44.127 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:32:44.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:44.128 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:44.322 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:45.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:45.323 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:45.323 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:32:45.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:32:45.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:45.324 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:45.519 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:46.520 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:46.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:46.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:32:46.521 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:32:46.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:46.521 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:46.719 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:47.720 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:32:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:32:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:47.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:48.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:48.921 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:48.921 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:32:48.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:32:48.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:48.922 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:49.118 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:50.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:50.119 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:50.120 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:32:50.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:32:50.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:50.120 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:50.318 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:51.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:51.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:51.320 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:32:51.320 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:32:51.320 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:51.320 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:51.513 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:52.514 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:52.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:52.515 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:32:52.515 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:32:52.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:52.516 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:52.708 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:53.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:53.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:53.711 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:32:53.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:32:53.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:53.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:53.907 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:54.908 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:54.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:54.909 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:32:54.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:32:54.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:54.910 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:55.106 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:56.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:56.107 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:56.107 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:32:56.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:32:56.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:56.108 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:56.302 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:57.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:57.304 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:57.304 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:32:57.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:32:57.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:57.305 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:57.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:58.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:58.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:58.499 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:32:58.499 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:32:58.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:58.500 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:58.697 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:32:59.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:32:59.699 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:32:59.699 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:32:59.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:32:59.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:32:59.700 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:32:59.917 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:00.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:00.918 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:00.919 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:33:00.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:33:00.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:00.919 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:33:01.113 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:02.114 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:02.115 INFO:tasks.workunit.client.0.smithi163.stdout:21 2024-05-06T04:33:02.115 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-06T04:33:02.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:02.116 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:33:02.314 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:03.315 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:03.316 INFO:tasks.workunit.client.0.smithi163.stdout:22 2024-05-06T04:33:03.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:03.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-05-06T04:33:03.316 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:03.317 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-06T04:33:03.591 INFO:tasks.workunit.client.0.smithi163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1223253570,v1:127.0.0.1:6801/1223253570] [v2:127.0.0.1:6802/1223253570,v1:127.0.0.1:6803/1223253570] exists,up cd1f0964-6e30-4b81-8960-e551aac65685 2024-05-06T04:33:03.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:33:03.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:33:03.600 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:33:03.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-05-06T04:33:03.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-05-06T04:33:03.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:33:03.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-06T04:33:03.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:33:03.601 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-05-06T04:33:03.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:33:03.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:33:03.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:33:03.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:33:03.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-05-06T04:33:03.602 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-05-06T04:33:03.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:33:03.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:33:03.603 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:33:03.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:33:03.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:33:03.603 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:33:03.604 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:33:03.604 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:33:03.604 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:33:03.604 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:33:03.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:33:03.604 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:33:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:33:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:33:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:33:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:33:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:33:03.605 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:33:03.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:33:03.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:33:03.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:33:03.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:33:03.606 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-05-06T04:33:03.607 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:33:03.607 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ab4d9a2-d925-4b83-9fe9-9fae92162dd0 2024-05-06T04:33:03.607 INFO:tasks.workunit.client.0.smithi163.stdout:add osd1 0ab4d9a2-d925-4b83-9fe9-9fae92162dd0 2024-05-06T04:33:03.608 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0ab4d9a2-d925-4b83-9fe9-9fae92162dd0' 2024-05-06T04:33:03.608 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:33:03.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/XThm7cBzJBAA/6pXtkg5xOZaa//0DAJvXQ== 2024-05-06T04:33:03.612 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/XThm7cBzJBAA/6pXtkg5xOZaa//0DAJvXQ=="}' 2024-05-06T04:33:03.613 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ab4d9a2-d925-4b83-9fe9-9fae92162dd0 -i td/ver-health/1/new.json 2024-05-06T04:33:04.150 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:33:04.160 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-05-06T04:33:04.161 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/XThm7cBzJBAA/6pXtkg5xOZaa//0DAJvXQ== --osd-uuid 0ab4d9a2-d925-4b83-9fe9-9fae92162dd0 2024-05-06T04:33:04.188 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.187+0000 7f6ac880f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:04.190 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.189+0000 7f6ac880f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:04.191 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.190+0000 7f6ac880f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:04.191 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.190+0000 7f6ac880f640 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-05-06T04:33:04.191 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.190+0000 7f6ac880f640 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-05-06T04:33:04.191 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.191+0000 7f6ac880f640 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-05-06T04:33:04.270 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:04.269+0000 7f6ac880f640 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-05-06T04:33:09.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-05-06T04:33:09.943 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:33:09.943 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd1 key to auth repository 2024-05-06T04:33:09.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-06T04:33:09.944 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:33:10.148 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-06T04:33:10.148 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.1 2024-05-06T04:33:10.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:33:10.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:33:10.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:33:10.149 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:33:10.174 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:10.173+0000 7f779ecae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:10.175 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:10.175+0000 7f779ecae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:10.176 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:10.176+0000 7f779ecae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:10.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-06T04:33:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:33:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-06T04:33:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:33:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:33:10.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:10.428 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:33:10.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:33:10.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:10.429 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:10.645 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:10.705 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:10.704+0000 7f779ecae640 -1 Falling back to public interface 2024-05-06T04:33:11.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:11.647 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:11.647 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:33:11.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:33:11.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:11.648 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:11.842 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:12.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:12.844 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:12.844 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:33:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:33:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:12.845 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:13.021 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:13.020+0000 7f779ecae640 -1 osd.1 0 log_to_monitors true 2024-05-06T04:33:13.041 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:14.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:14.042 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:14.043 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:33:14.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:33:14.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:14.043 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:14.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:15.237 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:15.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:15.238 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:33:15.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:33:15.238 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:15.239 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:15.437 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:16.438 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:16.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:16.439 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:33:16.439 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:33:16.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:16.440 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:16.634 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:17.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:17.635 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:17.636 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:33:17.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:33:17.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:17.636 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:17.832 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:18.834 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:18.834 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:33:18.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:18.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:33:18.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:18.835 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:19.031 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:20.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:20.033 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:20.033 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:33:20.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:33:20.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:20.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:20.230 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:21.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:21.231 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:21.231 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:33:21.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:33:21.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:21.232 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:21.426 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:22.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:22.427 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:22.427 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:33:22.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:33:22.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:22.428 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:22.622 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:23.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:23.624 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:33:23.624 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:23.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:33:23.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:23.625 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:23.821 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:24.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:24.823 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:24.823 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:33:24.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:33:24.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:24.824 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:25.021 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:26.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:26.023 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:26.023 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:33:26.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:33:26.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:26.024 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:26.222 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:27.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:27.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:27.224 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:33:27.224 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:33:27.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:27.225 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:27.420 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:28.421 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:28.422 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:33:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:33:28.422 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:28.423 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:28.617 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:29.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:29.618 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:29.618 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:33:29.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:33:29.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:29.619 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:29.815 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:30.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:30.817 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:30.817 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:33:30.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:33:30.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:30.818 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:31.287 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:32.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:32.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:32.289 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:33:32.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:33:32.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:32.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:32.482 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:33:33.484 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:33:33.484 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:33.485 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:33.681 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:34.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:34.682 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:34.682 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:33:34.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:33:34.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:34.683 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:34.880 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:35.881 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:35.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:35.882 INFO:tasks.workunit.client.0.smithi163.stdout:21 2024-05-06T04:33:35.882 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-06T04:33:35.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:35.883 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:36.163 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:37.164 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:37.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:37.165 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-05-06T04:33:37.165 INFO:tasks.workunit.client.0.smithi163.stdout:22 2024-05-06T04:33:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-06T04:33:37.382 INFO:tasks.workunit.client.0.smithi163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/508057246,v1:127.0.0.1:6809/508057246] [v2:127.0.0.1:6810/508057246,v1:127.0.0.1:6811/508057246] exists,up 0ab4d9a2-d925-4b83-9fe9-9fae92162dd0 2024-05-06T04:33:37.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:33:37.388 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-05-06T04:33:37.389 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-06T04:33:37.390 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee4b3e38-1429-49fc-ac7d-88d16b1698a4 2024-05-06T04:33:37.391 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ee4b3e38-1429-49fc-ac7d-88d16b1698a4' 2024-05-06T04:33:37.392 INFO:tasks.workunit.client.0.smithi163.stdout:add osd2 ee4b3e38-1429-49fc-ac7d-88d16b1698a4 2024-05-06T04:33:37.393 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-06T04:33:37.403 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChXThmjrn+FxAAAuDClh90iSfI9dqwlDY7NA== 2024-05-06T04:33:37.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChXThmjrn+FxAAAuDClh90iSfI9dqwlDY7NA=="}' 2024-05-06T04:33:37.404 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee4b3e38-1429-49fc-ac7d-88d16b1698a4 -i td/ver-health/2/new.json 2024-05-06T04:33:37.947 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:33:37.957 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-05-06T04:33:37.958 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChXThmjrn+FxAAAuDClh90iSfI9dqwlDY7NA== --osd-uuid ee4b3e38-1429-49fc-ac7d-88d16b1698a4 2024-05-06T04:33:37.985 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:37.983+0000 7f75e21a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:37.987 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:37.985+0000 7f75e21a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:37.988 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:37.986+0000 7f75e21a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:37.989 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:37.986+0000 7f75e21a3640 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-05-06T04:33:37.989 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:37.986+0000 7f75e21a3640 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-05-06T04:33:37.989 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:37.986+0000 7f75e21a3640 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-05-06T04:33:38.093 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:38.092+0000 7f75e21a3640 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-05-06T04:33:44.034 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-05-06T04:33:44.035 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-06T04:33:44.035 INFO:tasks.workunit.client.0.smithi163.stdout:adding osd2 key to auth repository 2024-05-06T04:33:44.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-06T04:33:44.036 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-06T04:33:44.242 INFO:tasks.workunit.client.0.smithi163.stdout:start osd.2 2024-05-06T04:33:44.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-06T04:33:44.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b61ee1d4-b4d6-43dd-ab31-04971d9d6f16 --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-06T04:33:44.243 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-06T04:33:44.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-06T04:33:44.244 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-06T04:33:44.268 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:44.267+0000 7feae4463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:44.270 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:44.269+0000 7feae4463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:44.271 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:44.270+0000 7feae4463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-06T04:33:44.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-06T04:33:44.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-06T04:33:44.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-06T04:33:44.528 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-06T04:33:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-06T04:33:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-06T04:33:44.529 INFO:tasks.workunit.client.0.smithi163.stdout:0 2024-05-06T04:33:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:44.529 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:44.754 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:44.801 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:44.800+0000 7feae4463640 -1 Falling back to public interface 2024-05-06T04:33:45.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:45.755 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:45.756 INFO:tasks.workunit.client.0.smithi163.stdout:1 2024-05-06T04:33:45.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-06T04:33:45.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:45.756 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:45.953 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:46.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:46.954 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:46.954 INFO:tasks.workunit.client.0.smithi163.stdout:2 2024-05-06T04:33:46.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-06T04:33:46.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:46.955 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:47.145 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:47.171 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:33:47.170+0000 7feae4463640 -1 osd.2 0 log_to_monitors true 2024-05-06T04:33:48.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:48.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:48.146 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-06T04:33:48.147 INFO:tasks.workunit.client.0.smithi163.stdout:3 2024-05-06T04:33:48.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:48.147 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:48.711 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:49.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:49.712 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:49.712 INFO:tasks.workunit.client.0.smithi163.stdout:4 2024-05-06T04:33:49.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-06T04:33:49.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:49.713 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:49.909 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:50.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:50.911 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:50.911 INFO:tasks.workunit.client.0.smithi163.stdout:5 2024-05-06T04:33:50.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-06T04:33:50.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:50.912 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:51.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:52.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:52.110 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:52.110 INFO:tasks.workunit.client.0.smithi163.stdout:6 2024-05-06T04:33:52.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-06T04:33:52.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:52.111 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:52.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:53.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:53.307 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:53.308 INFO:tasks.workunit.client.0.smithi163.stdout:7 2024-05-06T04:33:53.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-06T04:33:53.308 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:53.309 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:53.503 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:54.504 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:54.505 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:54.505 INFO:tasks.workunit.client.0.smithi163.stdout:8 2024-05-06T04:33:54.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-06T04:33:54.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:54.506 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:54.701 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:55.702 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:55.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:55.703 INFO:tasks.workunit.client.0.smithi163.stdout:9 2024-05-06T04:33:55.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-06T04:33:55.703 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:55.704 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:55.899 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:56.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:56.901 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:56.901 INFO:tasks.workunit.client.0.smithi163.stdout:10 2024-05-06T04:33:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-06T04:33:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:56.902 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:57.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:58.100 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:58.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:58.101 INFO:tasks.workunit.client.0.smithi163.stdout:11 2024-05-06T04:33:58.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-06T04:33:58.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:58.101 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:58.296 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:33:59.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:33:59.297 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:33:59.297 INFO:tasks.workunit.client.0.smithi163.stdout:12 2024-05-06T04:33:59.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-06T04:33:59.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:33:59.298 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:33:59.495 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:00.496 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:00.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:00.497 INFO:tasks.workunit.client.0.smithi163.stdout:13 2024-05-06T04:34:00.497 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-06T04:34:00.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:00.498 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:00.692 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:01.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:01.694 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:01.694 INFO:tasks.workunit.client.0.smithi163.stdout:14 2024-05-06T04:34:01.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-06T04:34:01.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:01.695 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:01.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:02.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:02.897 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:02.897 INFO:tasks.workunit.client.0.smithi163.stdout:15 2024-05-06T04:34:02.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-06T04:34:02.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:02.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:03.090 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:04.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:04.091 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:04.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-06T04:34:04.092 INFO:tasks.workunit.client.0.smithi163.stdout:16 2024-05-06T04:34:04.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:04.092 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:04.288 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:05.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:05.289 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:05.289 INFO:tasks.workunit.client.0.smithi163.stdout:17 2024-05-06T04:34:05.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-06T04:34:05.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:05.290 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:05.489 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:06.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:06.491 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:06.492 INFO:tasks.workunit.client.0.smithi163.stdout:18 2024-05-06T04:34:06.492 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-06T04:34:06.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:06.493 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:06.686 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:07.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:07.687 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:07.688 INFO:tasks.workunit.client.0.smithi163.stdout:19 2024-05-06T04:34:07.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-06T04:34:07.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:07.688 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:07.886 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:08.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:08.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:08.888 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-06T04:34:08.888 INFO:tasks.workunit.client.0.smithi163.stdout:20 2024-05-06T04:34:08.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:08.889 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:09.086 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-06T04:34:10.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-06T04:34:10.087 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-06T04:34:10.088 INFO:tasks.workunit.client.0.smithi163.stdout:21 2024-05-06T04:34:10.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-06T04:34:10.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-06T04:34:10.088 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-06T04:34:10.417 INFO:tasks.workunit.client.0.smithi163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1494961424,v1:127.0.0.1:6817/1494961424] [v2:127.0.0.1:6818/1494961424,v1:127.0.0.1:6819/1494961424] exists,up ee4b3e38-1429-49fc-ac7d-88d16b1698a4 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-05-06T04:34:10.419 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:11.029 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:34:11.030 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:34:11.030 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-05-06T04:34:11.080 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-06T04:34:11.304 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:34:11.305 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:34:11.306 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:34:11.343 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:34:11.344 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:34:11.344 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:11.344 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:11.344 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:34:11.344 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-05-06T04:34:11.379 INFO:tasks.workunit.client.0.smithi163.stdout:starting mds.m at 2024-05-06T04:34:11.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-05-06T04:34:11.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-05-06T04:34:11.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-05-06T04:34:11.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-05-06T04:34:11.384 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:11.385 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:34:11.386 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-05-06T04:34:11.450 INFO:tasks.workunit.client.0.smithi163.stdout:starting mds.n at 2024-05-06T04:34:11.453 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-05-06T04:34:16.455 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-05-06T04:34:17.138 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_OK 2024-05-06T04:34:17.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-05-06T04:34:17.150 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-05-06T04:34:17.770 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-05-06T04:34:17.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:34:17.771 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:34:17.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:34:17.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:34:17.771 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:34:17.875 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-05-06T04:34:17.876 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-05-06T04:34:17.895 INFO:tasks.workunit.client.0.smithi163.stderr:2024-05-06T04:34:17.894+0000 7f6a6ae47b00 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-05-06T04:34:17.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-06T04:34:17.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:34:17.897 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:34:17.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:34:17.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:17.897 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:17.898 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:34:17.898 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-05-06T04:34:18.098 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-06T04:34:18.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-05-06T04:34:18.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:34:18.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T04:34:18.099 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.b.asok 2024-05-06T04:34:18.100 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:34:18.101 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.b.asok config get fsid 2024-05-06T04:34:18.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-05-06T04:34:18.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-06T04:34:18.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-05-06T04:34:18.180 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.187931/ceph-mon.b.asok 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-06T04:34:18.181 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.187931/ceph-mon.b.asok config get mon_host 2024-05-06T04:34:18.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-05-06T04:34:18.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:34:18.259 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:34:18.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:34:18.259 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:34:18.260 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:34:18.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:34:18.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-05-06T04:34:18.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-05-06T04:34:18.365 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.187931/$cluster-$name.asok' 2024-05-06T04:34:18.366 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.187931/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-05-06T04:34:18.705 INFO:tasks.workunit.client.0.smithi163.stdout:starting mds.m at 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:34:18.710 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-05-06T04:34:19.205 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-05-06T04:34:20.206 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-05-06T04:34:20.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-05-06T04:34:20.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-05-06T04:34:20.207 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-05-06T04:34:20.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-05-06T04:34:20.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-05-06T04:34:20.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-05-06T04:34:20.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-05-06T04:34:20.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-05-06T04:34:20.773 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-05-06T04:34:21.339 INFO:tasks.workunit.client.0.smithi163.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-05-06T04:34:21.339 INFO:tasks.workunit.client.0.smithi163.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-05-06T04:34:21.339 INFO:tasks.workunit.client.0.smithi163.stdout: mon.b mds. are running an older version of ceph: 01.00.00-gversion-test 2024-05-06T04:34:21.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-05-06T04:34:21.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-05-06T04:34:21.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-05-06T04:34:21.979 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-05-06T04:34:22.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-05-06T04:34:22.615 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-05-06T04:34:23.235 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: return 1 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: return 1 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2332: main: code=1 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ver-health 1 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-06T04:34:23.236 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-06T04:34:23.345 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-06T04:34:23.345 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-06T04:34:23.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-06T04:34:23.346 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-06T04:34:23.346 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-06T04:34:23.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-06T04:34:23.347 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-06T04:34:23.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:34:23.347 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-06T04:34:23.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-06T04:34:23.348 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T04:34:23.348 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:34:23.349 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-05-06T04:34:23.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-05-06T04:34:23.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-05-06T04:34:23.350 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ver-health/mds.m.log td/ver-health/mds.n.log td/ver-health/mgr.x.log td/ver-health/mgr.y.log td/ver-health/mon.a.log td/ver-health/mon.b.log td/ver-health/osd.0.log td/ver-health/osd.1.log td/ver-health/osd.2.log /home/ubuntu/cephtest/archive/log 2024-05-06T04:34:23.351 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-05-06T04:34:23.360 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-06T04:34:23.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-06T04:34:23.361 INFO:tasks.workunit.client.0.smithi163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.187931 2024-05-06T04:34:23.361 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.187931 2024-05-06T04:34:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T04:34:23.362 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-06T04:34:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-06T04:34:23.363 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 1 2024-05-06T04:34:23.363 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-05-06T04:34:23.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-06T04:34:23.694 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/ver-health.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2024-05-06T04:34:23.946 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=00437f0fc6fd40a3b7dfa1537a7048c8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e08cbbf6fd79a8dd2949f340bce25e28ea3db004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/ver-health.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2024-05-06T04:34:23.949 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-06T04:34:23.961 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-06T04:34:23.961 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-06T04:34:24.014 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-06T04:34:24.023 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-06T04:34:24.023 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T04:34:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-05-06T04:34:24.048 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T04:34:24.048 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-05-06T04:34:24.048 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +624us[ +637us] +/- 50ms 2024-05-06T04:34:24.048 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -777us[ -777us] +/- 73ms 2024-05-06T04:34:24.048 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -142us[ -142us] +/- 59ms 2024-05-06T04:34:24.048 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T04:34:24.049 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-06T04:34:24.065 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-06T04:34:24.067 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-06T04:34:24.078 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T04:34:24.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-06T04:34:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T04:34:24.170 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-06T04:34:24.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-06T04:34:24.194 INFO:teuthology.task.internal:Duration was 3758.352575 seconds 2024-05-06T04:34:24.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-06T04:34:24.210 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-06T04:34:24.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T04:34:24.247 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T04:34:24.589 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-06T04:34:24.589 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-05-06T04:34:24.590 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T04:34:24.652 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-06T04:34:24.653 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T04:34:24.718 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-06T04:34:24.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T04:34:24.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-06T04:34:24.861 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-06T04:34:24.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T04:34:24.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-06T04:34:24.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:34:24.950 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-05-06T04:34:24.974 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T04:34:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T04:34:25.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-06T04:34:25.038 INFO:teuthology.task.internal:Transferring archived files... 2024-05-06T04:34:25.040 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi/7690146/remote/smithi163 2024-05-06T04:34:25.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T04:34:26.030 INFO:teuthology.task.internal:Removing archive directory... 2024-05-06T04:34:26.031 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T04:34:26.072 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-06T04:34:26.083 INFO:teuthology.task.internal:Not uploading archives. 2024-05-06T04:34:26.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-06T04:34:26.096 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-06T04:34:26.096 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T04:34:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 6 04:34 /home/ubuntu/cephtest 2024-05-06T04:34:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: 396938 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 6 03:37 /home/ubuntu/cephtest/mnt.0 2024-05-06T04:34:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: 396939 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 6 04:29 /home/ubuntu/cephtest/mnt.0/client.0 2024-05-06T04:34:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: 396982 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 6 04:29 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-06T04:34:26.128 INFO:teuthology.orchestra.run.smithi163.stdout: 411374 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 04:34 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-05-06T04:34:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T04:34:26.130 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-06T04:34:26.130 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-05-06T04:34:26.135 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-06T04:34:26.148 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-06T04:34:26.158 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi163 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-05-06T04:34:26.158 INFO:teuthology.nuke:Checking targets against current locks 2024-05-06T04:34:26.185 DEBUG:teuthology.nuke:shortname: smithi163 2024-05-06T04:34:26.186 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T04:34:26.211 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-03_23:44:50-rados-wip-yuriw-testing-20240503.181540-main-distro-default-smithi/7690146', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 03:25:24.627477', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvpA7ozbIen0YWnvxJtzldi8hpjW4oUapdfSZmK2PG8nBJSlU4oFVyMh9agfnL8WGEG9PnvTY3zz0hj1NmCinc='} 2024-05-06T04:34:26.273 INFO:teuthology.orchestra.console.smithi163:Power off 2024-05-06T04:34:26.274 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T04:34:26.305 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-05-06T04:34:26.306 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T04:34:26.335 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-05-06T04:34:30.336 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T04:34:30.448 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-05-06T04:34:34.450 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T04:34:34.560 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-05-06T04:34:34.661 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-05-06T04:34:34.770 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 3758.3525745868683 failure_reason: 'Command failed (workunit test misc/ver-health.sh) on smithi163 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e08cbbf6fd79a8dd2949f340bce25e28ea3db004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=00437f0fc6fd40a3b7dfa1537a7048c8 status: fail success: false 2024-05-06T04:34:34.770 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T04:34:34.849 INFO:teuthology.run:FAIL